Tools available for debugging production Issues in Java applications

Question

Hello,

Just wondering what are various tools & techniques out there to debug production issues on Java applications. Like,

• What are the ways and tools to take thread dumps?
• What are the ways and tools to take heap dumps?
• What are the tools to analyse the above dumps?

(Assumption all are in Linux/Unix environment)

Answer 1

Assuming JDK 6, Take a look at the following article to obtain thread dumps of a running program:

http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/

You can use JHat to do heap analysis:

http://java.sun.com/javase/6/docs/technotes/tools/share/jhat.html

If you want to do memory dumps take a look at jmap:

http://java.sun.com/javase/6/docs/technotes/tools/share/jmap.html

Alternatively, if you need to do some more in depth analysis look at a profiler such as Yourkit:

http://www.yourkit.com/

Answer 2

There's also these two things that can interest you:

• JConsole can dump the heap
• Ctrl-break to do a thread dump

Answer 3

And I think the best way to debug java application in production environment is NOT dump and so on, but a good managment of logs.

See for instance slf4j.

Answer 4

There is no standard toolset for JVMs. These are vendor dependant, and you should consult the documentation.

For Sun Java 6 the VisualVM program is very, very helpful to get a quick profile and stack trace of a running program.

Answer 5

What are the ways and tools to take thread dumps?

For a thread dump, you can use JConsole, VisualVM or, more simply, send a QUIT signal to the target process

kill -QUIT <pid> 

or

kill -3 <pid>

Since Java 5, there is also jstack which is platform independent and has a nice -m option to print both Java and native frames (mixed mode).

What are the ways and tools to take heap dumps?

With Sun VMs, jmap, Sun JConsole, Sun VisualVM, SAP JVMMon. For IBM VMs, check this page. Actually, the Eclipse MAT wiki has a nice Getting a Heap Dump section summarizing all the options.

What are the tools to analyse the above dumps?

For thread dumps I use TDA - Thread Dump Analyzer (for Sun JDKs) and IBM Thread and Monitor Dump Analyzer (for IBM JDKs). Samurai is also very nice (it works like a tail -f and picks up thread dumps from your std/stderr automatically, it can also read "-verbose:gc" logs) and has been tested against VMs from Apple, BEA, HP, Sun and IBM (can also read IBM's javacore).

For heap dumps, I use VisualVM (for Sun JDKs) or IBM Heap Dump Analyzer (only for IBM JDKs) or the über awesome Eclipse MAT depending on my needs. The later is able to work with HPROF binary heap dumps (produced by Sun, HP, SAP, etc... JVMs), IBM system dumps (after preprocessing them), and IBM portable heap dumps (PHD) from a variety of IBM platforms).

Answer 6

The tool I use for this kind of debugging a Sun JVM are

• jstack to take a thread dump
• jmap to take a memory/heap dump, or histogram
• eclipse mat for post analysis of the heap dump produced by jmap
• visual vm has a nice ui for live analysis of the vm (can also take heap and thread dumps)

Answer 7

To debug issues with memory allocations, InMemProfiler can be used at the command line. Live vs collected allocations can be tracked and collected objects can be split into buckets based on their lifetimes.

In trace mode this tool can be used to identify the source of memory allocations.