How to troubleshoot an unresponsive Java application/process in Linux

Question

Say your application is unresponsive and you cannot attach a debugger to it, as it rejects everything. All you have is a Linux Bash and process id. How would you investigate the issue? What tools would you use? My goal is to better my troubleshooting skills using Java.

This particular issue we had in production, on customer site.

Answer 1

You could take a thread dump from the application by issuing:

kill -3

That would give you some information as to the current state of the threads and hopefully help diagnose the issue. However, the trick is not in taking the thread dump, but reading the thread dump produced - since they can be a little overwhelming to look at. See this link for more info on reading a thread dump.

http://manikandakumar.blogspot.com/2006/12/reading-thread-dumps.html

You could also take a look at jstack which is part of the JDK - I've not used it specifically, see:

http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html

Answer 2

jstack <pid>

Answer 3

I agree with Jon that you should use kill -3 to get a thread dump. I have found Thread Dump Analyzer useful for viewing thread dumps.

You should also take a look at the memory usage of the process using top. Does it look like the app has run out of heap space? If so, you could try and use the jmap tool to obtain a heap dump and/or histogram count of the objects on the heap. You may need to use the -F option if the app has really hung up and I have experienced cases where jmap simply would not work against a hung Java process. Once you have a heap dump you could use Eclipse Memory Analyzer to investigate it.

You don't mention whether your application has any logging. If not you should look into adding logging that could help debug production issues.

Answer 4

Sounds like an interview question. You could also try attach jconsole to see what it is doing.

Answer 5

If you have Java 6 you can try to connect with Visualvm (https://visualvm.dev.java.net/) which ships with current JDKs to connect to the VM. With this Tool you are able to create a complete MemoryDump (not only thread dump) of your VM Process. You can load this Memory Dump into VisualVM or Eclipse with the MAT Plugin (Memory Analyzer Tools http://www.eclipse.org/mat/).

After some time of loading an computation you can browse the complete Heap of your Application, search form Memory Leaks etc.

Analysing Heap Dumps is a great way to improve your TroubleShooting Skills.

Answer 6

I agree with others that Thread dumps are the way to go. I would like to add that you should get lot's of thread dumps. You can do very simple profiling with just a few unix commands. Check my post here