How should I tackle with "Program Crashing" issues?

Question

Hi,

I am working on a Java product. An client claims that the application is getting crashed after an arbitrary time. SInce it is a crash we can't find any information on our logs.

1. Are there any tools, methods to find out the reasons for such Issues?

2. Can we do anything in code side to get more information on such program crashes?

3. Can we enable a "DEBUG" mode for JVM? If so where can I find the JVM log files/crash dumps?

4. Any known procedures to deal with this sort of issues?

5. If you got in to this problem, What would be your procedure in troubleshooting this?

Answer 1

After a crash, you don't have logs during the crash itself, but you still have all your logs before your actual crash. That should give you a lot of information, if your logs are detailed enough.

In java, you combine the two phases:

• logging in the code can be very detailed, using levels (fatal, error, warning, info, debug)
• logging can be configured in production to only output what is relevant only (even as specific as a single class's logs at debug level, while the rest is only at error level), to have a decent performance and log files of acceptable size.

Using the power of logging, you should be able to narrow your focus little by little. Note that, if your application has too few logs, you should start ASAP adding some more (at the appropriate logging levels of course). Example process:

1. activate error level for all the application, see what you get
2. activate warning level for one module, see what you get
3. deactivate the previous, activate info level for one package, see what you get
4. deactivate the previous, activate debug level for one class, see what you get

Answer 2

I find it hard to believe there's no output from the JVM when it crashes. Start by taking a long and hard look at your run scripts and seeing whether you are simply ignoring output. If the JVM ends due to an unhandled exception, it will output the exception to stdout I believe. If it crashes hard (heap corruption etc) it will output something to stderr. Your in-application logging is useful, but you should be logging any output that goes to stdout and stderr as well (you don't define the platform your app is running on, but this basically applies to all of them).

Aside from that, there's a whole host of non-standard options you can pass to define the location of error files and the like, see Java HotSpot VM Options.

Answer 3

At first yout should be aware, if the JVM crashes or your application itself. If your JVM crashes the java process creates several crash dumps on the file system, something like hs_errXXX.pid. If you find one of these files in the directory where java starts, you should check for this error on the official bug site at sun.

If your application crashes, you should extend your log infrastructure (like KLE mentioned). Using a shutdown hook to print out, that it is shut down (normally) is also quite handy .See here for API reference.

Answer 4

Hi,

I would adjust your application logging to verboser levels or tweak the JVM as pointed before, but if you want more options, you can try JVisualVM to watch something weird (memory/thread/gc/jmx operations) and, in the last chance, I would search for hs_err_pid*.log files. These files contains information about the state of the JVM in the moment of the hard crash (memory violations and so on). Here you have an example:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d741e3a, pid=1572, tid=1364
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# V  [jvm.dll+0x1e3a]
#

---------------  T H R E A D  ---------------

Current thread (0x00a85c78):  VMThread [id=1364]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000054

Registers:
EAX=0x00000050, EBX=0x00990000, ECX=0x0847b9f8, EDX=0x00000050
ESP=0x0ab0f660, EBP=0x0ab0f684, ESI=0x0847b9f8, EDI=0x0847b9f8
EIP=0x6d741e3a, EFLAGS=0x00010216

Answer 5

If you're using JNI (or any libraries that use JNI), it's easy to crash the JVM so that it leaves no traces at all. As far as I know, the only way to debug this kind of problems is to step through the native stuff with a debugger.

Answer 6

If this problem occurs only with that client, ask them if they run the application on more than one machine. If yes, does the problem occur on all of them?

If the problem occurs only on one machine, I'd suspect faulty hardware, most likely RAM. This can be diagnosed with a tool like memtest.

I've personally witnessed only two instances of recurring JVM crashes. In both cases, the problem was faulty RAM.

Answer 7

A few options that will help to diagnose memory issues:

The JVM option -XX:+HeapDumpOnOutOfMemoryError will create a heap dump if the VM exits due to memory exhaustion. You can analyse the dump using something like eclipseMAT to determine the cause of the problem.

Also -verbose:gc will provide detailed garbage collection stats, and adding -Xloggc:<file> will redirect this to a file.

Answer 8

In addition to all of the other suggestions, check your codebase for calls to System.exit().