debugging tomcat crash

Question

I have an instance of Tomcat which periodically crashes for unknown reasons.

There are no errors left in the logs, only a line in Event Viewer saying "Tomcat terminated unexpectedly".

In a test environment I have been unable to replicate the issue. I am therefore mostly restricted to passive monitoring of the production environment.

The problem does not seem to be related to memory as the unexpected terminations show no obvious correlation to the process' memory usage.

What steps could I take to further diagnose this problem?

EDIT:

Some corrections/clarifications:

It is actually not a single "instance" of Tomcat, rather several instances with similar configurations.

OS is Windows 2003.

Java version is Java 6.

UPDATE:

Looks like the issue might be related to memory after all. Discovered some crash dumps which were created in the Tomcat directory (not .../Tomcat/logs).

The dumps mostly contained errors such as:

java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

This is unexpected as the process sometimes crashed when it's memory usage was a relatively low point (compared to historical usage).

In all dumps, perm gen space is at 99% usage, but in absolute terms this usage is not consistent, and is nowhere near the limit specified in -XX:MaxPermSize.

Answer 1

This indicates to me that the whole JVM crashed, which is a rather unusual thing. I would consider the following steps:

• First check the hardware is ok. Run memtest86+ - http://www.memtest86.com/ or on a Ubuntu cd - to test the memory. Let it run a while to be absolutely certain.
• Then see if the version of Java you use, is ok. Some versions of Java 6 broke some subtle functionality. The latest Java 5 might be a good solution at this point.
• Disable the Tomcat native code to improve performance. There is a native library which Tomcat uses for something. Since you have a crashing JVM, getting rid of native code is a very good start.
• See if there is some restrictions in the version of Windows you use. A cpu usage limit before termination, or any other quota.

Answer 2

Generally if a process crashes in windows, a dump file is created. Load the dump file in windbg (windows debugger) and get a stack trace of the thread that caused the exception. This should give you a better idea what the problem is.