Open-Xchange Service stops randomly

[Markus Trinks]

Hello,

I'm new to this forum and hope my problem is placed correctly here.

Following problem:
We installed a new OXchange Server and bought all necessary licenses. After the installation we migrated all data from an pretty old OXchange. The old oxchange was installed on an old UCS 2.2.

The new Server has UCS version 4.0.3 and OXchange 7.6.2.

The migration went fine and the new server seems to run smoothly. But here comes the problem. Sometimes, and absolutly randomly the open-xchange service stops working. Because of that the OX Webinterface is not available and none of the 90 Users can access their accounts.

I looked into the syslog and open-xchange log files and the only error I could get is the following:

2015-10-03T14:37:32,407+0200 WARN  [OXWorker-0035722] org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:160)
Exception during FilterChain execution
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.obtainDirectByteBuffer(TCPNIOTransport.java:1207)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(TCPNIOTransport.java:830)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:74)
        at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:173)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:770)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$MDCProvidingRunnable.run(CustomThreadPoolExecutor.java:2498)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.runTask(CustomThreadPoolExecutor.java:813)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.run(CustomThreadPoolExecutor.java:840)
        at java.lang.Thread.run(Thread.java:745)
2015-10-03T14:37:32,659+0200 WARN  [OXWorker-0035722] org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:160)
Exception during FilterChain execution
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.obtainDirectByteBuffer(TCPNIOTransport.java:1207)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(TCPNIOTransport.java:830)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:74)
        at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:173)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:770)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$MDCProvidingRunnable.run(CustomThreadPoolExecutor.java:2498)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.runTask(CustomThreadPoolExecutor.java:813)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.run(CustomThreadPoolExecutor.java:840)
        at java.lang.Thread.run(Thread.java:745)

Restarting the open-xchange service solves the issue. Operating system is up to date. The problem already exists on UCS version 4.0.2. Restarting the server doesnt resolve the issue.

If you need any further information I will provide it. I'm not sure which informations you will need to have a closer look.


Thank you all!
Regards Markus Trinks

PS: Sorry for my poor english.

[Wolfgang Rosenauer]

If it's always out of memory what are your memory settings for the OX process?

/opt/open-xchange/etc/ox-scriptconf.sh
JAVA_XTRAOPTS
-Xmx
is the interesting portion.

[Markus Trinks]

Hello,

here the settings:

# Defines the Java options for the groupware Java virtual machine.
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC 
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB 
-XX:+DisableExplicitGC -Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow -Xmx1024m"

I guess the 1024 is the important value?

But as I said, I'm not sure if this error causes the crash. The logfiles are full with these error messages.

[Wolfgang Rosenauer]

1GB of memory for 90 (active) users sounds to small indeed.
I cannot guarantee that this is the only issue but for sure "out of memory" errors must not happen.
Can you increase the -Xmx to at least 1500 or 2048m and see if it helps?

[Markus Trinks]

Okay I changed it to 2048m. Do I need to restart any service to apply the change?

The machine has 4GB of RAM. It's a virtual Machine, so if more RAM is needed I can give the VM more RAM. Also I noticed that VMWare Tools are not installed. Should I install them?

Thank you very much for the help! I appreciate that.

[Wolfgang Rosenauer]

/etc/init.d/open-xchange restart is required indeed.
2GB for 90 users should be enough. If you need vmware tools or not I cannot easily answer. This is operating system ;-)

[Markus Trinks]

All right I restarted the service. The errors seems to be gone.

I will monitor the logs a few times a day. Hopefully that resolves the issue. If the service crashes again, I would like to share the event here in this thread.


Thank you for the help until now :-)

[Markus Trinks]

Hello again,

saturday the open-xchange webinterface was not reachable again. Following log entries appeared exactly at 03:30am.

(see attachement)

What does that mean? And is there a possiblilty to resolve the issue?


Thank you very much!


Regards
Markus Trinks

[Wolfgang Rosenauer]

So in general I would recommend to contact support in case you have the possibility to do so.

But the only issue I can find in the log is:
com.openexchange.exception.OXException: SES-0007 Categories=USER_INPUT Message='Max. number of sessions exceeded for user 89 in context 10' exceptionID=2002368882-4825

This does not explain "webinterface was not reachable". The above error only means that this user cannot login anymore with yet another session.

Please check mainly the sessiond.properties file for
com.openexchange.sessiond.maxSessionPerUser
com.openexchange.sessiond.maxSessionPerClient

Please note that the defaults are usually fine unless the user is doing something really strange.

The shutdown in the logs is hopefully something you did manually? Because there is no reason an OX server does a shutdown like this automatically.

[Markus Trinks]

Yes the shutdown was manually because there was no other way to make the Webinterface reachable again.

I will contact the support.
Appreciate the help and hope we get it fixed asap!

Thank you!