I have a production web application (Struts, iBatis, Hibernate) that runs in Tomcat that would hang while serving requests after 6 - 7 days of running but would run fine again after doing a thread dump.
I have a hard time figuring out why that is the case.
I was just wondering whether anyone else has ever encountered something similar.
First of all try to reproduce this in a test environment. You can use JMeter to stress the app. You can start tomcat using the -verbose:gc and -XX:+PrintGCDetails which will give you more insight on what is happening while GC runs. Then, when the site is not responding, you can get a thread dump and if this unblocks the site have a look at the GC details for more info.
Maybe this will help you find the cause of your problem.
I have enable JMX on tomcat (set these optional vm arguments when starting tomcat) -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=30188 (whatever port you want jmx to run on for tc) -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
I then wrote a little app that monitors memory usage (via jmx) and notifies me if the memory usage goes over , say 80%.
I would then know as soon as something is starting to go wrong. Then I will get a histogram for in-memory objects (see http://java.sun.com/javase/6/docs/technotes/tools/share/jmap.html for how to get that).
At the end it turned out that one of my ejbQL queries caused a huge amount of memory being used.
Hope it might help in some way ......