Homepage | Products | OX Knowledge Base | Support | Try Now | Contact | Company
OX Logo
Page 1 of 2 12 LastLast
Results 1 to 10 of 13
  1. #1
    Join Date
    Apr 2009
    Location
    Turin - Italy
    Posts
    68

    Default OXSE 6.12 - scrolling heavy Inbox sometimes throws ArrayIndexOutOfBoundsException

    I'm using latest OXSE 6.12 with Cyrus v2.3.14 and I'm testing system performance using a "heavy" user account (about 64000 unread messages in the Inbox folder).
    Sometimes, scrolling the Inbox folder with pageUp/pageDown keys leads to an ArrayIndexOutOfBoundsException and OX shows me a generic Error: 15 (MSG-9999, 1153503810-20).

    After some further test, I'm able to reproduce the problem with 2 different Error messages:

    Error: 15 (MSG-9999, 1153503810-20) (shown on OX client)

    Code:
    Sep 18, 2009 1:18:24 PM com.openexchange.ajax.Mail getWrappingOXException
    WARNING: An unexpected exception occurred, which is going to be wrapped for proper display.
    For safety reason its original content is display here.
    java.lang.ArrayIndexOutOfBoundsException: 15
            at com.openexchange.imap.IMAPCommandsCollection$21.doCommand(IMAPCommandsCollection.java:1541)
            at com.sun.mail.imap.IMAPFolder.doProtocolCommand(IMAPFolder.java:2616)
            at com.sun.mail.imap.IMAPFolder.doCommand(IMAPFolder.java:2566)
            at com.openexchange.imap.IMAPCommandsCollection.uids2SeqNums(IMAPCommandsCollection.java:1488)
            at com.openexchange.imap.IMAPMessageStorage.getMessagesLong(IMAPMessageStorage.java:236)
            at com.openexchange.mail.api.enhanced.MailMessageStorageLong.getMessages(MailMessageStorageLong.java:257)
            at com.openexchange.mail.MailServletInterfaceImpl.getMessageList(MailServletInterfaceImpl.java:707)
            at com.openexchange.ajax.Mail.actionPutMailList(Mail.java:2334)
            at com.openexchange.ajax.Mail.actionPutMailList(Mail.java:2258)
            at com.openexchange.ajax.request.MailRequest.action(MailRequest.java:156)
            at com.openexchange.ajax.Multiple.doAction(Multiple.java:289)
            at com.openexchange.ajax.Multiple.parseActionElement(Multiple.java:178)
            at com.openexchange.ajax.Multiple.doPut(Multiple.java:126)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:619)
            at com.openexchange.ajax.AJAXServlet.service(AJAXServlet.java:365)
            at com.openexchange.ajax.SessionServlet.service(SessionServlet.java:159)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
            at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.doServletService(AJPv13RequestHandlerImpl.java:433)
            at com.openexchange.ajp13.AJPv13Request.response(AJPv13Request.java:128)
            at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.createResponse(AJPv13RequestHandlerImpl.java:286)
            at com.openexchange.ajp13.najp.AJPv13ConnectionImpl.createResponse(AJPv13ConnectionImpl.java:189)
            at com.openexchange.ajp13.najp.AJPv13Task.run(AJPv13Task.java:281)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
            at java.util.concurrent.FutureTask.run(FutureTask.java:123)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
            at java.lang.Thread.run(Thread.java:595)
    Sep 18, 2009 1:18:24 PM com.openexchange.ajax.Mail actionPutMailList
    SEVERE: MSG-9999 Category=7 Message=15 exceptionID=304117002-8
    MSG-9999 Category=7 Message=15 exceptionID=304117002-8
            at com.openexchange.ajax.Mail.getWrappingOXException(Mail.java:210)
            at com.openexchange.ajax.Mail.actionPutMailList(Mail.java:2360)
            at com.openexchange.ajax.Mail.actionPutMailList(Mail.java:2258)
            at com.openexchange.ajax.request.MailRequest.action(MailRequest.java:156)
            at com.openexchange.ajax.Multiple.doAction(Multiple.java:289)
            at com.openexchange.ajax.Multiple.parseActionElement(Multiple.java:178)
            at com.openexchange.ajax.Multiple.doPut(Multiple.java:126)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:619)
            at com.openexchange.ajax.AJAXServlet.service(AJAXServlet.java:365)
            at com.openexchange.ajax.SessionServlet.service(SessionServlet.java:159)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
            at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.doServletService(AJPv13RequestHandlerImpl.java:433)
            at com.openexchange.ajp13.AJPv13Request.response(AJPv13Request.java:128)
            at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.createResponse(AJPv13RequestHandlerImpl.java:286)
            at com.openexchange.ajp13.najp.AJPv13ConnectionImpl.createResponse(AJPv13ConnectionImpl.java:189)
            at com.openexchange.ajp13.najp.AJPv13Task.run(AJPv13Task.java:281)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
            at java.util.concurrent.FutureTask.run(FutureTask.java:123)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
            at java.lang.Thread.run(Thread.java:595)
    Caused by: java.lang.ArrayIndexOutOfBoundsException: 15
            at com.openexchange.imap.IMAPCommandsCollection$21.doCommand(IMAPCommandsCollection.java:1541)
            at com.sun.mail.imap.IMAPFolder.doProtocolCommand(IMAPFolder.java:2616)
            at com.sun.mail.imap.IMAPFolder.doCommand(IMAPFolder.java:2566)
            at com.openexchange.imap.IMAPCommandsCollection.uids2SeqNums(IMAPCommandsCollection.java:1488)
            at com.openexchange.imap.IMAPMessageStorage.getMessagesLong(IMAPMessageStorage.java:236)
            at com.openexchange.mail.api.enhanced.MailMessageStorageLong.getMessages(MailMessageStorageLong.java:257)
            at com.openexchange.mail.MailServletInterfaceImpl.getMessageList(MailServletInterfaceImpl.java:707)
            at com.openexchange.ajax.Mail.actionPutMailList(Mail.java:2334)
            ... 20 more

    Error: [Not available] (MSG-9999, 1153503810-21) (shown on OX client)

    Code:
    Sep 21, 2009 11:40:21 AM com.openexchange.ajax.Mail getWrappingOXException
    WARNING: An unexpected exception occurred, which is going to be wrapped for proper display.
    For safety reason its original content is display here.
    java.lang.ArrayIndexOutOfBoundsException
    Sep 21, 2009 11:40:21 AM com.openexchange.ajax.Mail actionPutMailList
    SEVERE: MSG-9999 Category=7 Message=[Not available] exceptionID=1153503810-21
    MSG-9999 Category=7 Message=[Not available] exceptionID=1153503810-21
            at com.openexchange.ajax.Mail.getWrappingOXException(Mail.java:210)
            at com.openexchange.ajax.Mail.actionPutMailList(Mail.java:2360)
            at com.openexchange.ajax.Mail.actionPutMailList(Mail.java:2258)
            at com.openexchange.ajax.request.MailRequest.action(MailRequest.java:156)
            at com.openexchange.ajax.Multiple.doAction(Multiple.java:289)
            at com.openexchange.ajax.Multiple.parseActionElement(Multiple.java:178)
            at com.openexchange.ajax.Multiple.doPut(Multiple.java:126)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:619)
            at com.openexchange.ajax.AJAXServlet.service(AJAXServlet.java:365)
            at com.openexchange.ajax.SessionServlet.service(SessionServlet.java:159)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
            at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.doServletService(AJPv13RequestHandlerImpl.java:433)
            at com.openexchange.ajp13.AJPv13Request.response(AJPv13Request.java:128)
            at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.createResponse(AJPv13RequestHandlerImpl.java:286)
            at com.openexchange.ajp13.najp.AJPv13ConnectionImpl.createResponse(AJPv13ConnectionImpl.java:189)
            at com.openexchange.ajp13.najp.AJPv13Task.run(AJPv13Task.java:281)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
            at java.util.concurrent.FutureTask.run(FutureTask.java:123)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
            at java.lang.Thread.run(Thread.java:595)
    Caused by: java.lang.ArrayIndexOutOfBoundsException
    Sep 21, 2009 11:40:22 AM com.openexchange.ajax.Mail getWrappingOXException
    WARNING: An unexpected exception occurred, which is going to be wrapped for proper display.
    For safety reason its original content is display here.
    java.lang.ArrayIndexOutOfBoundsException
    Sep 21, 2009 11:40:22 AM com.openexchange.ajax.Mail actionPutMailList
    SEVERE: MSG-9999 Category=7 Message=[Not available] exceptionID=1153503810-22
    MSG-9999 Category=7 Message=[Not available] exceptionID=1153503810-22
            at com.openexchange.ajax.Mail.getWrappingOXException(Mail.java:210)
            at com.openexchange.ajax.Mail.actionPutMailList(Mail.java:2360)
            at com.openexchange.ajax.Mail.actionPutMailList(Mail.java:2258)
            at com.openexchange.ajax.request.MailRequest.action(MailRequest.java:156)
            at com.openexchange.ajax.Multiple.doAction(Multiple.java:289)
            at com.openexchange.ajax.Multiple.parseActionElement(Multiple.java:178)
            at com.openexchange.ajax.Multiple.doPut(Multiple.java:126)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:619)
            at com.openexchange.ajax.AJAXServlet.service(AJAXServlet.java:365)
            at com.openexchange.ajax.SessionServlet.service(SessionServlet.java:159)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
            at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.doServletService(AJPv13RequestHandlerImpl.java:433)
            at com.openexchange.ajp13.AJPv13Request.response(AJPv13Request.java:128)
            at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.createResponse(AJPv13RequestHandlerImpl.java:286)
            at com.openexchange.ajp13.najp.AJPv13ConnectionImpl.createResponse(AJPv13ConnectionImpl.java:189)
            at com.openexchange.ajp13.najp.AJPv13Task.run(AJPv13Task.java:281)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
            at java.util.concurrent.FutureTask.run(FutureTask.java:123)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
            at java.lang.Thread.run(Thread.java:595)
    Caused by: java.lang.ArrayIndexOutOfBoundsException

    No exception is thrown (at least for me) when using only the scrollbar to scroll the Inbox folder, so the reason for those exceptions could be a race condition between the scroll action (through keyboard) and the view selected email action (after a page scroll OX tries to load the currently selected email, which only occurs if I use keyboard to scroll up/down).


    Besides those scrolling exceptions, I experienced some performance problem when using heavy Inbox folder (with 64000 messages my browser hangs up for 8-10 seconds). I tried to tweak some OX settings (com.openexchange.imap.imapFastFetch and com.openexchange.mail.mailFetchLimit) but with no improvements, so I'm looking for some more tweaks.

  2. #2
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    3,695

    Default

    Hi,

    would you mind opening bug reports for those exceptions?

    Thanks a lot!

  3. #3
    Join Date
    Apr 2009
    Location
    Turin - Italy
    Posts
    68

    Default

    I opened a bug report here:
    https://bugs.open-xchange.com/show_bug.cgi?id=14544

    Any suggestion for improving email performance?

  4. #4
    Join Date
    Apr 2009
    Location
    Turin - Italy
    Posts
    68

    Exclamation network traffic question...

    After successfully applying your patch I'm trying to further investige how heavy inbox account can be somewhat improved.

    I found that, under some circumstances, OX generates much network traffic. For example, using the same 64000 Inbox messages, I noticed that my browser stops responding for many seconds while parsing JSON objects.
    In particular, the following ajax requests are sent to OX (/ajax/multiple?session=xxxx&continue=true) when clicking for the first time on the Inbox folder
    1. [{"action":"all","module":"mail","columns":"600,601 ","folder":"default0/INBOX","sort":"610","order":"desc"}]

      OX returns all email ids contained in the inbox folder:
      [{"data":[["65465","default0/INBOX"],["65466","default0/INBOX"],["65462","default0/INBOX"],...]]

    2. [{"action":"list","module":"mail","data":[{"folder":"default0/INBOX","id":"65465"},{"folder":"default0/INBOX","id":"65466"},...]

      OX returns the description for specified emails (the ones visible in the browser):
      [{"data":[["65465","default0/INBOX",3,false,32,[["root","root@srv-asgw02.csi.it"]],0,"Title 1",1253273253000,4291,0],["65466","default0/INBOX",3,false,0,[["root","root@srv-asgw02.csi.it"]],0,"Title 2",1253273253000,3290268,0],...]

    3. [{"action":"all","module":"mail","columns":"600,601","folder":"def ault0/INBOX","sort":"610","order":"desc"},{"action":"updates","module":"mail","timestamp":0,"columns":"600,601" ,"ignore":"deleted","folder":"default0/INBOX"}]

      Both a "list" and an "update" actions are issued and OX returns (another time ) the same exact response of the first query


    With 64000 emails, the first query returns about 1.75 MB of raw data, and the same data is also returned on step 3 (it's not clear to me why this step is required and why an update action is issued). So, when clicking on the inbox for the first time, about 1.75MB x 2 are transferred to the browser.

    When scrolling down the inbox, both step 2 and step 3 are repeated, so other 1.75 MB are transferred for each scroll


    Now my questions :
    • In step 1, is it possible to retrieve only a subset of the whole 64000 emails?
    • What is the purpose of step 3 "list" and "update" actions (repeated for every subsequent scroll)?
    • When querying for emails contained in a folder, why folder is repeated for every response object? Network traffic could be heavily reduced if folder is factored out and not repeated:

      [{"data":[["65465","default0/INBOX"],["65466","default0/INBOX"],["65462","default0/INBOX"],...]]
      ==>
      [{"data":[{"folder"="default0/INBOX", "data":["65465", "65466", "65462", ...]}]]


    Hope those observations can help you improve OX

  5. #5
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    3,695

    Default

    Hi,

    thanks for this report. We need all mail ids because the GUI needs to know how many mails are at the mailbox (to calculate how long the list needs to be) and what id they have since IDs are not consecutive. The GUI then generates the list and if a user clicks at position 20000 he requests the E-Mail with ID 4303 for example. This is also very important for sorting.
    Gonna forward this posting to get some more information.

    Greetings

  6. #6
    Join Date
    Apr 2009
    Location
    Turin - Italy
    Posts
    68

    Default

    thanks Martin for your near-realtime response
    I guess email ids are stored in some browser-side cache in order to fast scroll (as you pointed out, ids could be non-consecutive), and I think that this is a good thing for performance.

    The problem is that (in my opinion, of course ) when dealing with many messages, this query results in a very big JSON object and folderID is repeated for each object contained in the target folder...
    As folderID grows (nested folders, e.g: default0/INBOX/first subfolder/second subfolder), response size grows more and more

    I don't know why step 3 (from my previous post) occurs for each scroll, because email ids should be the same for each scroll request (at least since a refresh is required).


    I've also observed email fetch with more "normal" inboxes, and I've noticed that step 3 does not occur mostly time.
    When loading for the first time 1000 email, for example, both step 1 and 2 are issued and step 2 is repeated for each scroll (if not in cache, of course). Step 3 is never done...

  7. #7
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    3,695

    Default

    Hi,

    we located the problem. The main issue is, that the GUI object cache has a hardlimit of 20.000 entries. If those 20.000 entries which are cached at the client side are exceeded with mail IDs, the cache implementation starts to get all entries again which leads to performance issues. The traffic might not be the issues since it can be compressed using mod_deflate, but the client needs to compute stuff more often than required. This is a pretty nasty issue and not easy to fix. Generally the cache works well if you have three folders with 10000 mails each since the cache is cleared folder wise.

    As a temporary workaround, you could set the cache size to a higher value. This is done at the javascript code.

    OXCache.js:234:
    OXCache.objectSize = 20000;

    Setting this value to e.g. 100000 should solve your problem for now. Please report back if this had positive impact on performance. Since the cache is at the local client, the browser will eat up more memory so you might want to check what limit is acceptable. Please clear your browsers cache after applying this workaround.

    You also might want to set the autorefresh ("Reload current view every:") of the UI higher since this global value will fire requests every x minutes. While being at such a large folder it can also have negative impact.

    Greetings
    Last edited by Martin Heiland; 09-25-2009 at 05:53 PM.

  8. #8
    Join Date
    Apr 2009
    Location
    Turin - Italy
    Posts
    68

    Default

    Quote Originally Posted by Martin Braun View Post
    As a temporary workaround, you could set the cache size to a higher value. This is done at the javascript code.

    OXCache.js:234:
    OXCache.objectSize = 20000;

    Setting this value to e.g. 100000 should solve your problem for now.
    Just updated to 70000 and seems working fine
    Now, scrolling up/down my huge inbox does not issue step 3 and performance are very good. I also noticed that the top right refresh button does not auto-turn on during scrolling...is this related to the action=update in step 3?

    Quote Originally Posted by Martin Braun View Post
    The traffic might not be the issues since it can be compressed using mod_deflate, but the client needs to compute stuff more often than required.
    This could be a good workaround, but compressing the stream is quite an expensive task for both server and client. I would envisage a better handling of JSON data factoring out the folderId, but of course this activity requires refactoring the handling logic

    Thanks for your support!
    Last edited by janny_buh; 09-25-2009 at 06:37 PM.

  9. #9
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    3,695

    Default

    Hi,

    nice to hear that. The refresh button is animated on special requests like the 'all' or 'update' request. Since there should be no more requests like those (except when entering the folder first time) it won't be animated on scrolling anymore.
    Compressing might use some resources but in combination with mod_expires static data is not compressed on each request but only once every 24 hours for example. Compressing plain text like JSON objects is not such a big bottleneck. If the GUI runs without expires and deflate it gets very slow. Even installations with several million mailboxes work well with ssl+deflate+expires. You might enable those modules and monitor the resource usage.
    I already opened a internal bug report for this issue. We have several options, here some examples:

    1. Set the cache size to a higher value by default (e.g. 100000)
    * This would only shift the problem until someone with 100001 mails uses the software and would consume more memory at the client for all users, even for those with 100 folders with 1000 objects each but no large folder.

    2. Separate the ID and the content list where the ID list can grow unlimited
    * We had that already in the past but in terms of making the cache more "simplistic" (it's acually pretty complex even though) we put all kinds of data to one cache region.

    3. Dynamically extend the cache size temporary if a single folder with more than 20000 objects is requested.
    * This might be the optimal solution but of course the most complex one.

    4. < insert your idea here >


    Have a nice weekend!

  10. #10
    Join Date
    Apr 2009
    Location
    Turin - Italy
    Posts
    68

    Default

    Quote Originally Posted by Martin Braun View Post
    Compressing might use some resources but in combination with mod_expires static data is not compressed on each request but only once every 24 hours for example. Compressing plain text like JSON objects is not such a big bottleneck. If the GUI runs without expires and deflate it gets very slow.
    Just enabled mod_deflate on my Apache and there is a big improvement in network bandwidth. My original 1.75MB JSON object is reduced to 170KB, which is of course a good news , but I've few questions:
    • Are there any side effect when enabling mod_deflate with JSON queries? I noticed that with the default installation from OX tutorial, JSON queries are not compressed by default (while html and js are compressed) and I need to manually add "text/javascript" to the AddOutputFilterByType section...
    • Enabling mod_deflate helps keeping network traffic between client and frontend Apache down, but I think it has no effect for network traffic between Apache and OX backend (when OX and Apache are not on the same host, of course), isn't it?


    As regards OXCache, I'm not enough skilled in OX engine and for my actual installation the 1st option fits my needs, but for the future the 3rd option seems the most appropriate

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •