Why does my Tomcat/Alfresco site become unresponsive every half a day or so?

Question

I'm a developer running a website on a CentOS 5 box with Apache Tomcat 6.x, Alfresco Labs 3 (the standalone .war file) and a couple custom webapps.

The server has been up and running without problem for the past several months, but just the other day someone was uploading an image through Alfresco and the site stopped responding. In addition to that, all the webapps running through Tomcat were unresponsive as well. I tried telnetting in locally (i.e. telnet localhost 80) but it timed out without connecting.

Looking in the most recent log file (localhost...log) I saw a stack trace starting with java.lang.OutOfMemoryError: PermGen space failure and after some googling increased the max perm size from 128MB to 384MB. We thought that this would fix the problem, but today it has come back and tomcat is down again (except without the PermGen stack trace in the logs)

The other stack trace that keeps popping up is the following:

Jun 16, 2009 3:38:26 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet default threw exception
java.lang.IllegalStateException
    at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:407)
    at org.apache.struts2.dispatcher.Dispatcher.sendError(Dispatcher.java:707)
    at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:485)
    at org.apache.struts2.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java:395)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
    at java.lang.Thread.run(Thread.java:619)

I can't tell if this is related, as it doesn't seem to actually crash Tomcat, but it seems pop up fairly often. One other reason I don't think it is a big deal is that has been showing up in our logs for the past month or so.

Other than that, I'm not entirely sure what to do. There aren't really any other clues I know to check. I'm guessing that this is an Alfresco issue, but I can't say for sure. The only thing I did today to change the configuration was increase the permgen size to 512MB, but that may not help since we didn't actually get the PermGen error this time.

Any ideas?

Answer 1

the easiest fix may be to check to see whether the version of tomcat you are using has this error in any other versions, you may just need to upgrade/revert tomcat.

found this , hope its useful

http://www.nabble.com/-jira--Created:-(WW-3106)-SEVERE:-Servlet.service()-for-servlet-default-threw-exception-java.lang.IllegalStateException-(Shows-a-BLANK-PAGE-to-user)-td23291783.html

Answer 2

Up from 256 to 512 MB stabilized reoccuring tomcat permgen instabilities here. Double the RAM guarantees ad-hoc improvement and very probably gets situation from fixes to development. Otherwise lose time profiling, get know why, and anyhow later double the RAM.

Answer 3

You shold also look for a memory leak -- are you loading classes from the war that could be loaded from the shared library? Also -- certain logging classes can cause tomcat to hold on to loaded classes ( which is what fills the permgen space ) from being GC'd. Look into tomcat class-loaders and make sure you're not loading more than you need to from the application level class loaders.

Answer 4

Otherwise lose time profiling, get know why, and anyhow later double the RAM.