Global Address Book is Extremely slow

[masikh]

Dear Open-Xchange Users and Developers,

At the University of Leiden (the Netherlands) we are looking for a replacement of Netmail (novell). The two products we are looking at are Zarafa and Open-Xchange. One of the demands of our new e-mail (groupware) system is the availability of a GAL (Global Adress List). Now here comes the crux: We have at least 40.000 users in one GAL. These are students, employees and others. If we look-up a user in say MS-Exchange (yes we use that too!) it's really snappy (response time is less than 1/10 Seconds) but unfortunately the same cannot be said about Open-Xchange. The response time is ~180 seconds, furthermore the process-load of the java process shoots to a admirable 200% CPU-time. The load on the (other TIER) mysql-server is a mere 17% (nose picking!). The mysql-server is optimized for caching and it looks it behaves just as such. So where is this 200% CPU-time coming from. As you can clearly see, a CPU-time of 200% for a single user looking up a GAL is unacceptable, let alone 5000 (concurrent!!!) users are doing this. And yes they will do this as soon as they log in. Personally I'd like Open-Xchange to be the winner of Zarafa thus I'm willing to spend a lot of time into this problem but I am stuck. The settings I tried to tweak (and succeeded to some extend!) are:

/opt/open-xchange/etc/admindaemon/ox-admin-scriptconf.sh:

JAVA_XTRAOPTS="-Xms4096m -Xmx4096m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=0 \
-XX:SurvivorRatio=20000 -XX:+UseCMSInitiatingOccupancyOnly \
-XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -Djava.net.preferIPv4Stack=true"

JAVA_OXCMD_OPTS="-Xms4096m -Xmx4096m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=0 \
-XX:SurvivorRatio=20000 -XX:+UseCMSInitiatingOccupancyOnly \
-XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -Djava.net.preferIPv4Stack=true"

AND

/opt/open-xchange/etc/groupware/ox-scriptconf.sh:

JAVA_XTRAOPTS="-Xmx4096m -Xms4096m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=0 \
-XX:SurvivorRatio=20000 -XX:+UseCMSInitiatingOccupancyOnly \
-XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC"

These setting result in a first time login of say 3 minutes. I assume it's building some sort of cache???. The second time it will be in the order of a minute. By login-time I mean, the amount of time the JAVA process takes to process all thing it must process to let a user login. For the user (no being able to see the Linux-terminal) it is the time needed before the Global Address List is viewable. If I scroll a bit through this GAL the whole thing starts over, and it typically takes about a minute to show results.

As you can see, this will not scale up to 40.000 users let alone the 5000 concurrent users we are about to expect, leaving Zarafa the clear winner (I don't like that).

Futhermore, I we do a bulk import (via oxldapsync, the horror!) it process 1 user every 3 seconds?. The load of java shoots up and so on...

So what's to be done? Any ideas?

Kind regards,

Robert Nagtegaal.

some info about the software and machines:

MACHINE 0: LVS (linux virtual machine): Load balance between the four ox servers.
MACHINE 1: OX+Apache+dovecot+postfix: SLES 11.1, 64bit, 6GB working memory, Net-Apps FibreChannel disks.
MACHINE 2: MySQL cluster (preformance is good)
MACHINE 3: LDAP cluster (preformance is good)

[Martin Heiland]

Hi Robert,

there is a lot stuff that can be configured and tuned when deploying OX in large environments, especially when dealing with many users within one context. I suggest to get in touch with our professional services people for a proof-of-concept installation and consulting in dealing with larger installations. The global address book can be limited to be search-only for example and several settings at the database might be optimized. Some large universities already use OX successfully so there should be a solution for that.

Greetings

[masikh]

Dear Martin Braun,

Personally I'd love to do that, but this project is NOT allowed to cost any money (at least in this stadium), thus jumping into the consultancy band-wagon isn't a viable option at ~this~ time. But maybe I can contact the university who has dealt with these issues, any pointers? Isn't there any one in the community who has a user-base of this magnitude?

Kind regards,

Robert Nagtegaal.

[Daniel Halbe]

Dear Robert,

please send me your contact data to daniel.halbe (AT) open-xchange.com After this I will contact you next week.

It is in general not usefull to put 40k users into one tenant - Open-Xchange is multi tenant capable so use this feature

Best regards,
Daniel

[masikh]

Dear Daniel,

As requested I have sended my e-mail adress to you. True, in general it is not usefull to put 40K users in one tenant, but we are talking about a Univerity Wide Adress Book. Thus a subdivision into logical tenants (e.g. a faculty) is not going to work. A tenant is an isolated groupware instance. They each have their own adress book. Thus a user allways get a subset of the whole University GAL, which isn't the goal of this project.

Meanwhile I finished my import of users (41919) after three days. If I have a single login on a OX machine, java shoots to 200% and stays that way for 10 minutes and more. If I am lucky my session will live during this time. The GAL, btw, is not usable at all due to its slowness. The MySQL servers shoot to 38% utilization but as soon everything is in memory mysql is running normal again.

Any pointers for optimalization (on a very short notice!) are very welcome. Next monday, they (FB) are going to look at both products (zarafa and Open Xchange) and make a choice. Being able to just use the product would be nice. So, please help me out!

Kind regards,

Robert.

[masikh]

Most unfortunately I had to shut-down the GAL. This severely cripples the OX installation with respect to usability. Are there any pointers as how to solve the above sketched situation? There are other installations with the same amount of users, do they have the same problems? Is there some secret magic code only available in the commercial version making the GAL run on steroids?

Again, any pointers are welcome,

Kind regards,

Robert Nagtegaal.

[Martin Heiland]

Hi,

there is a tweaks for the GAL etc/groupware/contact.properties:

# Determines if a search is triggered if the dialog for searching for emailable
# contacts is opened. This dialog is used for selecting recipients for an email
# and for creating distribution lists.
com.openexchange.contact.mailAddressAutoSearch=true

This should be switched to "false" to avoid long loading times when opening the "recipient" or "participant" dialogs. Basically folders with 40k entries are not such a big issue since their content is not loaded by the UI entirely, only chunk-wise when scrolling through the folder.

Can you describe in what situation the load/wait time gets up? While logging in, generally or when executing certain tasks? Are there any entries at the logfiles available while these long waiting times occur? This would help for sure to locate possible bottlenecks.

Setting 4GB of RAM to the JAVA_OXCMD_OPTS and JAVA_XTRAOPTS at the admindaemon/ox-scriptconf.sh is not required, these settings are for the command line tools (100MB should be enough) and for the admindaemon (512MB should do). 4GB for the groupware/ox-scriptconf.sh is okay for the groupware process but you might not need those additional parameters. We suggest about 4MB memory per logged in user. Of course, Java will consume all of the allowed memory when required. I doubt that 40k users are using the system concurrently and we have installations with about 8k concurrent users running on machines with 16GB of memory. Did you already checked out the Sizing Whitepaper? (http://software.open-xchange.com/OX6...whitepaper.pdf) we also have some best practice pages at the wiki (http://oxpedia.org/wiki/index.php?ti..._Tutorial_100K)

Greetings

[masikh]

Dear Martin Braun,

It seems the newer release 6.20.0 Rev5 does not suffer from any sluggyness. It still has high demands on memory and databases but it works more or less. I am trying to export my current database of users (~43900) and reuse it in a new setup from scratch. Personally I think I tweaked to much, thereby ruining the performance. We will see. I hope I return to this place with answers instead of questions.

Many thanks for all the help thus far,

Robert Nagtegaal.