Announcement

Collapse
No announcement yet.

Unable to log into groupware app - variety of error messages

Collapse
X
Collapse
 
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
Previous template Next
  • #1

    Unable to log into groupware app - variety of error messages

    Hi,

    Have been unable to log into groupware web interface.
    Attempts usually produce the following error - "503 - service temporarily unavailable"
    Though occasionally the error is one of the following instead...

    "500 - internal server error"
    "Error: Cannot get connection to database. (CTX-0005, -618270995-371)"
    "07-08-2010 12:28 AM-->Error: A socket error occurred: Too many open files (MSG-1021, 618471929-2272)"

    Rebooting the server always fixes the problem, but only temporarily. After another day or so, the problem recurs and cannot log in until the server is rebooted again. Accordingly suspected that maybe a service that launches upon boot, is getting messed up or getting shutdown while the server is running. So tried manually starting each service that automatically launches on startup, and a few other things, but nothing helped.
    This is what was attempted:

    1) stopped and started the groupware
    2) Ran all these commands...

    a2enmod proxy
    a2enmod proxy_ajp
    a2enmod expires
    a2enmod deflate
    a2enmod headers

    insserv mysql
    insserv apache2
    insserv open-xchange-groupware
    insserv open-xchange-admin

    checking the log files did not show anything conspicuous, but maybe the wrong ones were checked.

    Searching through the forum returned too many threads complaining about "503 - service temporarily unavailable", so instead of an inexperienced user trying to figure out which thread is relevant, could you point out which one(s) might help with this problem and/or give some guidance?

    Linux OpenSuse 11.2
    OX Version : 6.16.1 Rev3 ( 2010-04-23 11:06:42)

    Thank you
    Last edited by Guest; 07-28-2010, 05:44 AM.
    Tags: None
  • #2
    Hi,

    seems like your server runs out of resources. Is it some kind of VPS with limited resources? Are there monitoring information about memory consumption and open file descriptors?

    Greetings
    Last edited by Martin Heiland; 07-28-2010, 09:15 AM.

    Comment

    • #3
      Hi,

      System resources seem to be more than sufficient and OX ran fine for awhile before this problem began. There have been no changes/reduction in system resources at any point.

      System details:
      Dual P3 1ghz processors, 1GB ram with 75% consumed, swap partition is 1.5GB, which is barely used, hard disc is 32GB with 24GB free, set up as a unified file system - 1 partition.

      After further troubleshooting, it seems that AJP is the cause of the issue. When OX is running fine, AJP13 is running also on the open port 8009/TCP.
      But when the problem occurs, AJP13 has stopped running. So the question now is what would be stopping AJP?

      We are NOT running tomcat. Other than the lightly loaded web server, we do not run any other server applications.

      We found some suspicious entries in the logs...

      These errors are scattered throughout the file /var/log/localmessages, but a cluster of them were at the end of the file:

      Code:
      Aug  1 12:20:10 GrandCentral checkproc: checkproc: can not get session id for process 4021!
Aug  1 13:42:17 GrandCentral checkproc: checkproc: can not get session id for process 6313!
Aug  1 17:36:23 GrandCentral checkproc: checkproc: can not get session id for process 6003!
Aug  1 19:37:50 GrandCentral checkproc: checkproc: can not get session id for process 7823!
Aug  1 20:12:17 GrandCentral checkproc: checkproc: can not get session id for process 10892!
Aug  1 22:14:40 GrandCentral checkproc: checkproc: can not get session id for process 13279!
Aug  2 21:51:05 GrandCentral checkproc: checkproc: can not get session id for process 12479!
Aug  2 23:10:42 GrandCentral checkproc: checkproc: can not get session id for process 16089!
      In open-xchange-console.log:

      Code:
      Could not find fetcher.properties on classpath
java.net.SocketException: Too many open files
    at java.net.Socket.createImpl(Socket.java:414)
    at java.net.Socket.connect(Socket.java:544)
    at java.net.Socket.connect(Socket.java:495)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:174)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:409)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:240)
    at sun.net.www.http.HttpClient.New(HttpClient.java:321)
    at sun.net.www.http.HttpClient.New(HttpClient.java:338)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:814)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:755)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:680)
    at com.openexchange.subscribe.crawler.internal.CrawlerUpdateTask.run(CrawlerUpdateTask.java:104)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$ScheduledFutureTask.runPeriodic(CustomThreadPoolExecutor.java:856)
    at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$ScheduledFutureTask.run(CustomThreadPoolExecutor.java:881)
    at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.runTask(CustomThreadPoolExecutor.java:737)
    at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.run(CustomThreadPoolExecutor.java:763)
    at java.lang.Thread.run(Thread.java:636)

      In open-xchange.log.3:

      Code:
      Aug 2, 2010 10:51:26 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem calling the UDP Discovery Sender [224.0.0.1:6781]
java.net.SocketException: Too many open files
	at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:69)
	at java.net.DatagramSocket.createImpl(DatagramSocket.java:319)
	at java.net.DatagramSocket.<init>(DatagramSocket.java:227)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:159)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:105)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender.<init>(UDPDiscoverySender.java:67)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:150)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:51:26 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem closing Passive Broadcast sender
java.lang.NullPointerException
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:171)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:51:56 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender <init>
SEVERE: Could not bind to multicast address [224.0.0.1]
java.net.SocketException: Too many open files
	at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:69)
	at java.net.DatagramSocket.createImpl(DatagramSocket.java:319)
	at java.net.DatagramSocket.<init>(DatagramSocket.java:227)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:159)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:105)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender.<init>(UDPDiscoverySender.java:67)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:150)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:51:56 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem calling the UDP Discovery Sender [224.0.0.1:6780]
java.net.SocketException: Too many open files
	at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:69)
	at java.net.DatagramSocket.createImpl(DatagramSocket.java:319)
	at java.net.DatagramSocket.<init>(DatagramSocket.java:227)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:159)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:105)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender.<init>(UDPDiscoverySender.java:67)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:150)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:51:56 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem closing Passive Broadcast sender
java.lang.NullPointerException
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:171)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:51:56 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender <init>
SEVERE: Could not bind to multicast address [224.0.0.1]
java.net.SocketException: Too many open files
	at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:69)
	at java.net.DatagramSocket.createImpl(DatagramSocket.java:319)
	at java.net.DatagramSocket.<init>(DatagramSocket.java:227)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:159)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:105)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender.<init>(UDPDiscoverySender.java:67)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:150)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:51:56 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem calling the UDP Discovery Sender [224.0.0.1:6781]
java.net.SocketException: Too many open files
	at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:69)
	at java.net.DatagramSocket.createImpl(DatagramSocket.java:319)
	at java.net.DatagramSocket.<init>(DatagramSocket.java:227)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:159)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:105)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender.<init>(UDPDiscoverySender.java:67)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:150)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:51:56 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem closing Passive Broadcast sender
java.lang.NullPointerException
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:171)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:52:26 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender <init>
SEVERE: Could not bind to multicast address [224.0.0.1]
java.net.SocketException: Too many open files
	at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:69)
	at java.net.DatagramSocket.createImpl(DatagramSocket.java:319)
	at java.net.DatagramSocket.<init>(DatagramSocket.java:227)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:159)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:105)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender.<init>(UDPDiscoverySender.java:67)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:150)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:52:26 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem calling the UDP Discovery Sender [224.0.0.1:6780]
java.net.SocketException: Too many open files
	at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:69)
	at java.net.DatagramSocket.createImpl(DatagramSocket.java:319)
	at java.net.DatagramSocket.<init>(DatagramSocket.java:227)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:159)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:105)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender.<init>(UDPDiscoverySender.java:67)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:150)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:52:26 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem closing Passive Broadcast sender
java.lang.NullPointerException
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:171)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:52:26 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender <init>
SEVERE: Could not bind to multicast address [224.0.0.1]
java.net.SocketException: Too many open files
	at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:69)
	at java.net.DatagramSocket.createImpl(DatagramSocket.java:319)
	at java.net.DatagramSocket.<init>(DatagramSocket.java:227)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:159)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:105)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender.<init>(UDPDiscoverySender.java:67)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:150)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:52:26 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem calling the UDP Discovery Sender [224.0.0.1:6781]
java.net.SocketException: Too many open files
	at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:69)
	at java.net.DatagramSocket.createImpl(DatagramSocket.java:319)
	at java.net.DatagramSocket.<init>(DatagramSocket.java:227)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:159)
	at java.net.MulticastSocket.<init>(MulticastSocket.java:105)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySender.<init>(UDPDiscoverySender.java:67)
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:150)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
Aug 2, 2010 10:52:26 AM org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread run
SEVERE: Problem closing Passive Broadcast sender
java.lang.NullPointerException
	at org.apache.jcs.auxiliary.lateral.socket.tcp.discovery.UDPDiscoverySenderThread.run(UDPDiscoverySenderThread.java:171)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
	at java.lang.Thread.run(Thread.java:636)
      Since we are not running in a clustered environment, we changed sessioncache.ccf and cache.ccf based on our impression from the error log files...

      In sessioncache.ccf, we changed several values from true to false:

      19,20c19,20
      < jcs.region.SessionCache.elementattributes.IsRemote =true
      < jcs.region.SessionCache.elementattributes.IsLatera l=true
      ---
      > jcs.region.SessionCache.elementattributes.IsRemote =false
      > jcs.region.SessionCache.elementattributes.IsLatera l=false
      32c32
      < jcs.auxiliary.SessionLTCP.attributes.UdpDiscoveryE nabled=true
      ---
      > jcs.auxiliary.SessionLTCP.attributes.UdpDiscoveryE nabled=false
      34c34
      < jcs.auxiliary.SessionLTCP.attributes.Receive=true
      ---
      > jcs.auxiliary.SessionLTCP.attributes.Receive=false


      In cache.ccf, we also changed some values from true to false.

      16c16
      < jcs.default.elementattributes.IsLateral=true
      ---
      > jcs.default.elementattributes.IsLateral=false
      279c279
      < jcs.auxiliary.LTCP.attributes.Receive=true
      ---
      > jcs.auxiliary.LTCP.attributes.Receive=false
      282c282
      < jcs.auxiliary.LTCP.attributes.IssueRemoveOnPut=tru e
      ---
      > jcs.auxiliary.LTCP.attributes.IssueRemoveOnPut=fal se


      The problem usually recurs within a day, so we should soon have an idea if this worked. However, we're in a rush to settle this issue conclusively, and would like your input regardless. What are your thoughts?

      Thank you

      Comment

      • #4
        Hi Martin,

        The problem has recurred. Everything else works well enough, but this one issue makes the product essentially unusable. We're pressed for time to get this working again. Do you have any ideas what might be causing this?

        Thank you

        Comment

        Previous template Next
        Working...
        X