If you have a Java application that is consuming CPU when it isn't doing anything how do you determine what it is doing?

Question

I am calling a vendors Java API and on some servers it appears that the JVM goes into a low priority polling loop after logging into the API (cpu 100%). The same app on other servers does not exhibit this behavior. This happens on WebSphere and Tomcat. The environment is tricky to setup so it is difficult to try to do something like profiling within eclipse. Is there a way to profile (or some other method of inspecting) an existing Java app running in Tomcat to find out what methods are being executed while it's in this spinwait kind of state? The app is only executing one method when it gets in this state (vendor's method). Vendor can't duplicate (of course). Thanks in advance.

Update:

Using JConsole I was able to determine who was running and what they were doing. It took me a few hours to then figure out why it was doing it. The solution ended up being the vendors api jar that was being used did not match exactly to the the database configuration that it was using. It was defaulting to having tracing and performance monitoring enabled on the servers that had the slight mis-match in configuration. I used a different jar and all is well.

So thanks Joshua, for your answer. JConsole was extremely easy to setup and monitor an existing application.

@Cringe - I did some experimenting with some of the options you suggested. I had some problems with getting JProfiler setup, it looks good (but pricey). Going forward I went ahead and added the Eclipse Profiler plugin and I'll be looking over the different open source profilers to compare functionality. Thanks.

Answer 1

If you are using Java 5 or later, you can connect to your application using jconsole to view all running threads. jstack also will do a stack dump. I think this should still work even inside a container like Tomcat.

Both of these tools are included with JDK5 and later (I assume the process needs to be at least Java 5, though I could be wrong)

Update: It's also worth noting that starting with JDK 1.6 update 7 there is now a bundled profiler called VisualVM which can be launched with 'jvisualvm'. It looks like it is a java.net project, so additional info may be available at that page. I haven't used this yet but it looks useful for more serious analysis.

Hope that helps

Answer 2

If it's for professional purpose and you have some money to spend, try to get your hands on JProfiler. If you just want to get some insights, try out the Eclipse Profiler Plugin. I used it several times, but I don't know the current state.

A new(?) project from the eclipse project itself is available too: http://www.eclipse.org/tptp/ (See this article). Never used it, so I can't tell if it is worth the effort.

There's also a very good list of open source profilers available at http://www.manageability.org/blog/stuff/open-source-profilers-for-java

Answer 3

Facing the same problem I used YourKit profiler. It's loader doesn't activate unless you actually connect to it (though it does open a port to listen for connections). The profiler itself has a nice "get amount of time spent in each method" while working in it's less obtrusive mode.

Another way is to detect CPU load (via JNI, so you'd need an external library for this) in a "watchdog" thread with highest priority and start logging all threads when the CPU is high enough for a long enough time. You might find this article enlightining.

Answer 4

For completeness sake: even though my company more or less standardizes on Eclipse we use Netbeans (6 and up) with its included, free profiler on a daily basis. It works better than the Eclipse TPTP plugin (last checked 3 months ago) and for us it removes any need for a commercial profiler such as JProfiler, which is excellent, but fast becoming unnecessary.

Answer 5

Bruce, please see my related question (more importantly, the answers...)

Answer 6

VisualVM should be the profiler from netbeans as standalone. I tried the TPTP for eclipse but visualVm seems as a much nicer option!

Answer 7

Use a profiler. Yes they cost money, and using them can occasionally be a bit awkward, but they do provide you with a great deal more real evidence rather than guesswork.

Human beings are universally bad at guessing where performance bottlenecks are. It just seems to be something our brains aren't build to do very well. It may seem obvious, you may have great ideas about what the problem is, but the real world often turns out to be doing something different. And optimising the wrong part of code means, at best, lots of work for minimal benefit. More often it makes things slower, and sometimes it breaks things entirely. So before you make any changes for the sake of optimisation, you should always have real evidence from a profiler or other accurate tool.

As mentioned, both JProfiler and YourKit are both fairly good and not prohibitively expensive. Last time I looked, they both had free demos too.

Answer 8

If JConsole can't be used you can

• press CTRL+BREAK under Windows
• send kill -3 <process id> under Linux

to get a full Thread Dump. This doesn't affect performance and can always be run in production.

Answer 9

JRockit Mission Control Latency Analyzer.

The Latency Analyzer that comes with JRockit shows you what the JVM is "doing" when it's not doing anything. In the latest version you can see latencies for:

• Java wait/blocked/sleep/parked.
• File I/O
• Network I/O
• Memory allocation
• GC pauses
• JVM latencies, e.g code generation and class loading
• Thread suspension

The tool will give you the stack trace when the latency occurred. You can view the latency data in many different ways (aggregated traces, as a histogram, in a thread graph etc.). The tool also allows you to see transitions between threads, for instance when one thread notifies another.

The overhead is negligible and unlike many other tools it can be used in a production environment. This blog post gives you a brief introduction and the program can be downloaded here.

It's free to use for development!