Automatically detect hanging threads

Question

I'm currently looking for a way to identify hanging threads in java. Anyone knows whats the best way to do this?

Currently I could think off 2 possible ways to do this:

1. Calling a (callback-)method periodically within all methods of the Application. This seems a "bit" complex and unsightly... Moreover I have no control when calling external methods...
2. An additional Thread which periodically generates Thread-dumps for all Threads (or maybe just for some which should be monitored - as I know which thread I want to monitor) and analyzes the result ("is the thread still at the same point with locks on the same objects" , ...). This could be a bit dangerous as the Thread may again(!) be at the same point... By the way - Is there an easy way to get the Thread dump within Java 1.4 (I don't want to call an external application). I guess with 1.5 or 1.6 there are methods to easily do this...

I guess non of these two methods is a good solution... So do you know a way to do this?

As I said before: I don't want to use any external applications...

Answer 1

If you are looking for an "automatic" way. There is no good solution except to spend more time in the design process. If you have threads that are "hanging" then they haven't been closed properly. I would spend time looking at how to contextually close them rather than trying to write a routine/thread that looks for stale threads.

Answer 2

This is just an idea:

Keep in a centralized (synchronized) class information about the starting time of a Thread and a reference to the thread. In example a ConcurrentHashMap (from now CHM).

Past some (conservative) time you can ask to CHM for that Thread (or watever thread you suspect it could be hanged) for it's starting time. If the thread is still alive you can be (conservatively) sure that it's hanged.

You can go further and keep the stacktrace of the threads, as commented before.

Answer 3

I've found that when we have threads that are stuck, that we usually end up with a performance problem in our application server.

We have a low tech way of trying to determine where the threads are stuck.

We send several kill -3 signal to the JVM to generate several thread dumps, and then analyse the output looking for similar traces, indicating problematic code.

Low tech and manual, but it worked.

Answer 4

The question here is what "hanging thread" means and in what context.

Is inside a big J2EE server (which could probably find out itself) or a tiny Java program where you spawn tons of threads? If it is the latter, consider investigating the concurrent utilities in Java 5+ which allows you to have Runnables and Callables under control.

Answer 5

If you have a race condition, then bringing additional resources (i.e. installing a JDK and using jps and jstack) to bear on the problem is surely reasonable...

That said, if you are running an app outside of a container, you can just run the app from the console and hit ctrl+break (Windows) to get the thread dump. Or a kill -3 on *nix.

But based on your answers in comments, it sounds like this is probably running in an app server of some sort.

The problem here is that pretty much anything you do is going to introduce further uncertainty to the situation. How will you know that your deadlock monitor isn't itself responsible for deadlocks?

Another option would be to introduce logging of system inputs, then create a re-play of what happened in the system based on those inputs. If your customer has basically said that they aren't going to help you troubleshoot your race condition, then you need to find it yourself in your own test environment. Emulating the load conditions of the customer site is one way of attempting this.

But really, I recommend that you work with them to get a JDK installed (so you can use jstack), or a JRE 1.4 monitoring application of some sort (heck, make sure that their app server doesn't already provide this sort of thing).