OXSE 6.12 - scrolling heavy Inbox sometimes throws ArrayIndexOutOfBoundsException

**Martin Heiland** · 09-21-2009, 01:32 PM

Hi,

would you mind opening bug reports for those exceptions?

Thanks a lot!

**janny_buh** · 09-21-2009, 02:48 PM

I opened a bug report here:

https://bugs.open-xchange.com/show_bug.cgi?id=14544

Any suggestion for improving email performance?

**janny_buh** · 09-25-2009, 04:39 PM

network traffic question...

After successfully applying your patch I'm trying to further investige how heavy inbox account can be somewhat improved.

I found that, under some circumstances, OX generates much network traffic. For example, using the same 64000 Inbox messages, I noticed that my browser stops responding for many seconds while parsing JSON objects.
In particular, the following ajax requests are sent to OX (/ajax/multiple?session=xxxx&continue=true) when clicking for the first time on the Inbox folder

[{"action":"all","module":"mail","columns":"600,601 ","folder":"default0/INBOX","sort":"610","order":"desc"}]

OX returns all email ids contained in the inbox folder:
[{"data":[["65465","default0/INBOX"],["65466","default0/INBOX"],["65462","default0/INBOX"],...]]
[{"action":"list","module":"mail","data":[{"folder":"default0/INBOX","id":"65465"},{"folder":"default0/INBOX","id":"65466"},...]

OX returns the description for specified emails (the ones visible in the browser):
[{"data":[["65465","default0/INBOX",3,false,32,[["root","root@srv-asgw02.csi.it"]],0,"Title 1",1253273253000,4291,0],["65466","default0/INBOX",3,false,0,[["root","root@srv-asgw02.csi.it"]],0,"Title 2",1253273253000,3290268,0],...]
[{"action":"all","module":"mail","columns":"600,601","folder":"def ault0/INBOX","sort":"610","order":"desc"},{"action":"updates","module":"mail","timestamp":0,"columns":"600,601" ,"ignore":"deleted","folder":"default0/INBOX"}]

Both a "list" and an "update" actions are issued and OX returns (another time ) the same exact response of the first query

With 64000 emails, the first query returns about 1.75 MB of raw data, and the same data is also returned on step 3 (it's not clear to me why this step is required and why an update action is issued). So, when clicking on the inbox for the first time, about 1.75MB x 2 are transferred to the browser.

When scrolling down the inbox, both step 2 and step 3 are repeated, so other 1.75 MB are transferred for each scroll

Now my questions

:

In step 1, is it possible to retrieve only a subset of the whole 64000 emails?
What is the purpose of step 3 "list" and "update" actions (repeated for every subsequent scroll)?
When querying for emails contained in a folder, why folder is repeated for every response object? Network traffic could be heavily reduced if folder is factored out and not repeated:

[{"data":[["65465","default0/INBOX"],["65466","default0/INBOX"],["65462","default0/INBOX"],...]]
==>
[{"data":[{"folder"="default0/INBOX", "data":["65465", "65466", "65462", ...]}]]

Hope those observations can help you improve OX

**Martin Heiland** · 09-25-2009, 05:03 PM

Hi,

thanks for this report. We need all mail ids because the GUI needs to know how many mails are at the mailbox (to calculate how long the list needs to be) and what id they have since IDs are not consecutive. The GUI then generates the list and if a user clicks at position 20000 he requests the E-Mail with ID 4303 for example. This is also very important for sorting.
Gonna forward this posting to get some more information.

Greetings

**janny_buh** · 09-25-2009, 05:45 PM

thanks Martin for your near-realtime response

I guess email ids are stored in some browser-side cache in order to fast scroll (as you pointed out, ids could be non-consecutive), and I think that this is a good thing for performance.

The problem is that (in my opinion, of course

) when dealing with many messages, this query results in a very big JSON object and folderID is repeated for each object contained in the target folder...
As folderID grows (nested folders, e.g: default0/INBOX/first subfolder/second subfolder), response size grows more and more

I don't know why step 3 (from my previous post) occurs for each scroll, because email ids should be the same for each scroll request (at least since a refresh is required).

I've also observed email fetch with more "normal" inboxes, and I've noticed that step 3 does not occur mostly time.
When loading for the first time 1000 email, for example, both step 1 and 2 are issued and step 2 is repeated for each scroll (if not in cache, of course). Step 3 is never done...

**Martin Heiland** · 09-25-2009, 05:51 PM

Hi,

we located the problem. The main issue is, that the GUI object cache has a hardlimit of 20.000 entries. If those 20.000 entries which are cached at the client side are exceeded with mail IDs, the cache implementation starts to get all entries again which leads to performance issues. The traffic might not be the issues since it can be compressed using mod_deflate, but the client needs to compute stuff more often than required. This is a pretty nasty issue and not easy to fix. Generally the cache works well if you have three folders with 10000 mails each since the cache is cleared folder wise.

As a temporary workaround, you could set the cache size to a higher value. This is done at the javascript code.

OXCache.js:234:
OXCache.objectSize = 20000;

Setting this value to e.g. 100000 should solve your problem for now. Please report back if this had positive impact on performance. Since the cache is at the local client, the browser will eat up more memory so you might want to check what limit is acceptable. Please clear your browsers cache after applying this workaround.

You also might want to set the autorefresh ("Reload current view every:") of the UI higher since this global value will fire requests every x minutes. While being at such a large folder it can also have negative impact.

Greetings

**janny_buh** · 09-25-2009, 06:36 PM

Originally posted by Martin Braun View Post

As a temporary workaround, you could set the cache size to a higher value. This is done at the javascript code.

OXCache.js:234:
OXCache.objectSize = 20000;

Setting this value to e.g. 100000 should solve your problem for now.

Just updated to 70000 and seems working fine

Now, scrolling up/down my huge inbox does not issue step 3 and performance are very good. I also noticed that the top right refresh button does not auto-turn on during scrolling...is this related to the action=update in step 3?

Originally posted by Martin Braun View Post

The traffic might not be the issues since it can be compressed using mod_deflate, but the client needs to compute stuff more often than required.

This could be a good workaround, but compressing the stream is quite an expensive task for both server and client. I would envisage a better handling of JSON data factoring out the folderId, but of course this activity requires refactoring the handling logic

Thanks for your support!

**Martin Heiland** · 09-25-2009, 06:48 PM

Hi,

nice to hear that. The refresh button is animated on special requests like the 'all' or 'update' request. Since there should be no more requests like those (except when entering the folder first time) it won't be animated on scrolling anymore.
Compressing might use some resources but in combination with mod_expires static data is not compressed on each request but only once every 24 hours for example. Compressing plain text like JSON objects is not such a big bottleneck. If the GUI runs without expires and deflate it gets very slow. Even installations with several million mailboxes work well with ssl+deflate+expires. You might enable those modules and monitor the resource usage.
I already opened a internal bug report for this issue. We have several options, here some examples:

1. Set the cache size to a higher value by default (e.g. 100000)
* This would only shift the problem until someone with 100001 mails uses the software and would consume more memory at the client for all users, even for those with 100 folders with 1000 objects each but no large folder.

2. Separate the ID and the content list where the ID list can grow unlimited
* We had that already in the past but in terms of making the cache more "simplistic" (it's acually pretty complex even though) we put all kinds of data to one cache region.

3. Dynamically extend the cache size temporary if a single folder with more than 20000 objects is requested.
* This might be the optimal solution but of course the most complex one.

4. < insert your idea here >

Have a nice weekend!

**janny_buh** · 09-28-2009, 12:23 PM

Originally posted by Martin Braun View Post

Compressing might use some resources but in combination with mod_expires static data is not compressed on each request but only once every 24 hours for example. Compressing plain text like JSON objects is not such a big bottleneck. If the GUI runs without expires and deflate it gets very slow.

Just enabled mod_deflate on my Apache and there is a big improvement in network bandwidth. My original 1.75MB JSON object is reduced to 170KB, which is of course a good news

, but I've few questions:

Are there any side effect when enabling mod_deflate with JSON queries? I noticed that with the default installation from OX tutorial, JSON queries are not compressed by default (while html and js are compressed) and I need to manually add "text/javascript" to the AddOutputFilterByType section...
Enabling mod_deflate helps keeping network traffic between client and frontend Apache down, but I think it has no effect for network traffic between Apache and OX backend (when OX and Apache are not on the same host, of course), isn't it?

As regards OXCache, I'm not enough skilled in OX engine and for my actual installation the 1st option fits my needs, but for the future the 3rd option seems the most appropriate

**Martin Heiland** · 09-28-2009, 02:32 PM

Hi,

we have had some bad experience with mod_deflate+mod_ssl+mod_jk in the early times so we don't compress the json requests by default. You're free to check if it works properly with the latest version of apache.
mod_deflate only compresses the traffic between the client (browser) and the server (apache). The traffic between apache and OX is ajp13 which is a binary format that is not handled by mod_deflate. AFAIK there is a way to compress ajp13 too, but i never tried it. Typically bandwidth for ajp calls is not an issue at the backend.

Greetings

**janny_buh** · 09-28-2009, 06:15 PM

Originally posted by Martin Braun View Post

we have had some bad experience with mod_deflate+mod_ssl+mod_jk in the early times so we don't compress the json requests by default.

My company uses Apache 2.0.61, do you know of some problem with this version and mod_deflate+mod_ssl+mod_jk?

Looking back at OXCache improvement, would be possible to move the Inbox folder into a new subfolder once a month? Such a mechanism (employed by Horde webmail for example) should prevent the Inbox folder to became too much overloaded

**Martin Heiland** · 09-28-2009, 06:35 PM

We're proposing mod_proxy_ajp instead of mod_jk now. The problem was that the performance was very bad and even session timed out due to data corruption by mod_jk. I don't have the feeling that this issue still occurs with proxy_ajp. Automatically moving E-Mail is a nice feature, you might open an enhancement bug for it. Thanks!

OXSE 6.12 - scrolling heavy Inbox sometimes throws ArrayIndexOutOfBoundsException

OXSE 6.12 - scrolling heavy Inbox sometimes throws ArrayIndexOutOfBoundsException

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment