Performance / System hangs

[jefigo]

Hello,
periodically our ox server hangs with 100% cpu usage. We have increased the log level to debug.

The lines from the open-xchange.log:

FINER: JMX.mbean.registered com.openexchange.pooling:name=DB Pool 4
May 3, 2011 4:48:53 PM com.openexchange.sessiond.impl.SessionHandler cleanUp
INFO: Session timed out. ID: 8e892bbf5b4a416499199ed1a0aa30bf
May 3, 2011 4:48:53 PM com.openexchange.sessiond.impl.SessionHandler cleanUp
INFO: Session timed out. ID: 3ef6d66f60204f999b70b149e9f1a76e
May 3, 2011 4:48:53 PM com.openexchange.sessiond.impl.SessionHandler cleanUp
INFO: Session timed out. ID: cd84d78984244ba2b81fd3eb0e0836ee
May 3, 2011 4:48:53 PM com.openexchange.mail.cache.MailAccessCacheEventListener handleEvent
INFO: Detected a removed session: cd84d78984244ba2b81fd3eb0e0836ee. Removed all possibly cached mail access instances for user 29 in context 1
May 3, 2011 4:48:53 PM com.openexchange.mail.cache.MailAccessCacheEventListener handleEvent
INFO: Detected a removed session: 8e892bbf5b4a416499199ed1a0aa30bf. Removed all possibly cached mail access instances for user 36 in context 1
May 3, 2011 4:48:53 PM com.openexchange.mail.cache.MailAccessCacheEventListener handleEvent
INFO: Detected a removed session: 3ef6d66f60204f999b70b149e9f1a76e. Removed all possibly cached mail access instances for user 54 in context 1
May 3, 2011 4:50:15 PM com.openexchange.ajp13.najp.AJPv13TaskWatcher$TaskRunCallable handleExceededTask
SEVERE: AJP KEEP-ALIVE failed.
AJP-0004 Category=6 Message=No data provided from web server: input stream returned "-1" while reading AJP magic bytes in package #2. Wait for input data took 0msec. exceptionID=1101825611-143229
        at com.openexchange.ajp13.najp.AJPv13ConnectionImpl.readInitialBytes(AJPv13ConnectionImpl.java:579)
        at com.openexchange.ajp13.najp.AJPv13TaskWatcher$TaskRunCallable.keepAliveGetEmptyBody(AJPv13TaskWatcher.java:440)
        at com.openexchange.ajp13.najp.AJPv13TaskWatcher$TaskRunCallable.keepAlive(AJPv13TaskWatcher.java:404)
        at com.openexchange.ajp13.najp.AJPv13TaskWatcher$TaskRunCallable.handleExceededTask(AJPv13TaskWatcher.java:357)
        at com.openexchange.ajp13.najp.AJPv13TaskWatcher$TaskRunCallable.call(AJPv13TaskWatcher.java:330)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.runTask(CustomThreadPoolExecutor.java:750)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.run(CustomThreadPoolExecutor.java:776)
        at java.lang.Thread.run(Thread.java:662)
May 3, 2011 4:50:20 PM com.openexchange.ajp13.AJPv13ServletOutputStream responseToWebServer
WARNING: Underlying (TCP) protocol communication aborted: Broken pipe
java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at com.openexchange.ajp13.BlockableBufferedOutputStream.flushBuffer(BlockableBufferedOutputStream.java:126)
        at com.openexchange.ajp13.BlockableBufferedOutputStream.flush(BlockableBufferedOutputStream.java:184)
        at com.openexchange.ajp13.AJPv13ServletOutputStream.responseToWebServer(AJPv13ServletOutputStream.java:292)
        at com.openexchange.ajp13.AJPv13ServletOutputStream.flush2WebServer(AJPv13ServletOutputStream.java:264)
        at com.openexchange.ajp13.AJPv13ServletOutputStream.flush(AJPv13ServletOutputStream.java:112)
        at com.openexchange.ajp13.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:186)
        at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.doResponseFlush(AJPv13RequestHandlerImpl.java:543)
        at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.doServletService(AJPv13RequestHandlerImpl.java:529)
        at com.openexchange.ajp13.AJPv13Request.response(AJPv13Request.java:134)
        at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.createResponse(AJPv13RequestHandlerImpl.java:335)
        at com.openexchange.ajp13.najp.AJPv13ConnectionImpl.createResponse(AJPv13ConnectionImpl.java:227)
        at com.openexchange.ajp13.najp.AJPv13Task.call(AJPv13Task.java:352)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.runTask(CustomThreadPoolExecutor.java:750)
        at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.run(CustomThreadPoolExecutor.java:776)
        at java.lang.Thread.run(Thread.java:662)
May 3, 2011 4:50:20 PM com.openexchange.ajp13.najp.AJPv13Task call
SEVERE: AJP-0016 Category=6 Message=An I/O error occurred: Broken pipe exceptionID=1101825611-144254

During the peak the system is nearly standing. Even on the command line nothing is possible.

Any idea?

Regards
Jens

[Martin Heiland]

Hi Jens,

this seems like a connection issue between OX and Apache (mod_proxy_ajp). Please also check the apache log files for the given timeframe to get more information. Are there any subsequent error messages at the OX log regarding permission issues? For example timeouts or OutOfMemory messages. Feel free to contact the support as well if this causes issues to a production system.

Greetings