views:

2221

answers:

9

Let's say your Java program is taking 100% CPU. It has 50 threads. You need to find which thread is guilty. I have not found a tool that can help. Currently I use the following very time consuming routine:

  1. Run jstack <pid>, where pid is the process id of a Java process. The easy way to find it is to run another utility included in the JDK - jps. It is better to redirect jstack's output to a file.
  2. Search for "runnable" threads. Skip those that wait on a socket (for some reason they are still marked runnable).
  3. Repeat steps 1 and 2 a couple of times and see if you can locate a pattern.

Alternatively, you could attach to a Java process in Eclipse and try to suspend threads one by one, until you hit the one that hogs CPU. On a one-CPU machine, you might need to first reduce the Java process's priority to be able to move around. Even then, Eclipse often isn't able to attach to a running process due to a timeout.

I would have expected Sun's visualvm tool to do this.

Does anybody know of a better way?

A: 

This is a kind of hacky way, but it seems like you could fire the application up in a debugger, and then suspend all the threads, and go through the code and find out which one isn't blocking on a lock or an I/O call in some kind of loop. Or is this like what you've already tried?

Paul Fisher
yep, this is what I was doing more or less. But this is quite tedious, especially if there is many threads and when on a one-CPU machine, which is the case of a small EC2 instance and when you run in VirtualBox.
Gene Vayngrib
+8  A: 

Just run up JVisualVM, connect to your app and and use the thread view. The one which remains continually active is your most likely culprit.

Software Monkey
Even with JConsole(bundled with Java 1.5 and up) you can do the same.
adrian.tarau
JVisualVM is bundled with J6 too.
Software Monkey
Java 6 update 10 and later.
Thorbjørn Ravn Andersen
Any commercial Java profiler application will do the same, if for some reason you are stuck on an earlier version of the JDK.
Bill Michell
@Bill: Good point. I have personally used JProfiler for years with code from Java 1 on and found it to work very well.
Software Monkey
Thank you. JVisualVM indeed shows threads as runnable or not. But that does not mean they are actually running, some may be sitting on a socket - not sure why they are still marked as runnable - note that some threads are always green in JVisualVM threads view. So it may give you an idea but not a definite answer to what thread is hogging cpu.
Gene Vayngrib
Yes, it seems that the JVM considers any thread in a native method other than wait() to be running. It's annoying, but it still seems easy to eliminate them to find the real culprit.
Software Monkey
+1  A: 

If you're running under Windows, try Process Explorer. Bring up the properties dialog for your process, then select the Threads tab.

jdigital
We just used this approach for locating a Thread loop and it worked fine. The Eclipse debug view is great for starting and stopping threads. You can pause all of them and then reactivate them one by one to see which one is spinning.
Jim Ferrans
+4  A: 

Try looking at the Hot Thread Detector plugin for visual VM -- it uses the ThreadMXBean API to take multiple CPU consumption samples to find the most active threads. It's based on a command-line equivalent from Bruce Chapman which might also be useful.

Cowan
I tried Bruce Chapman's command line Hot Thread detector and it worked for me (it needed some class from tools.jar so I had to copy tools.jar to jre/lib/ext on my Ubuntu Jaunty (9.04) with Sun Java 6 update 13). I prefer a command line tool in this case since system is very usually so slow that GUI tools are a real pain.
Gene Vayngrib
A: 

Take a thread dump. Wait for 10 seconds. Take another thread dump. Repeat one more time. Inspect the thread dumps and see which threads are stuck at the same place, or processing the same request. This is a manual way of doing it, but often useful.

talonx
A: 

An option you could consider is querying your threads for the answer from within application. Via the ThreadMXBean you can query CPU usage of threads from within your Java application and query stack traces of the offending thread(s).

The ThreadMXBean option allows you to build this kind of monitoring into your live application. It has negligible impact and has the distinct advantage that you can make it do exactly what you want.

Neil Coffey
A: 

If you suspect VisualVM is a good tool, try it (because it does this) Find out the threads(s) only helps you in the general direction of why it is consuming so much CPU.

However, if its that obvious I would go straight to using a profiler to find out why you are consuming so much CPU.

Peter Lawrey
A: 

Are you running Java 6 on a multi-core computer?

Chances are that you are suffering from a problem I just described in an article about thread starvation.

See Synchronized vs. Lock vs. fair Lock.

Huxi
+1  A: 

Have a look at the Top Threads plugin for JConsole.

Mark
thank you, next time I get into the 100% cpu I will compare the Bruce's command line tool and this plugin
Gene Vayngrib