Does this stack dump indicate that I have a deadlock ?

Question

I have a REST service built using Jersey.

When I performed a "curl" against my REST API, the command hangs.

I ran jstack & this is a summarized output of two threads in BLOCKED state.

"pool-2-thread-11" prio=6 tid=0x01d51800 nid=0x2394 
 waiting for monitor entry [0x05e6f000..0x05e6fce8]
java.lang.Thread.State: BLOCKED (on object monitor)
    at com.moi.DefaultImageProcessor$DownloadAndScaleCallable.call(
          DefaultImageProcessor.java:168)
    - waiting to lock <0x257aa440> 
     (com.moi.ImageUriMutexImpl$MutexImpl)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
        ThreadPoolExecutor.java:885)

"pool-2-thread-10" prio=6 tid=0x01d51000 nid=0x18d0
 waiting for monitor entry [0x05e1f000..0x05e1fd68]
java.lang.Thread.State: BLOCKED (on object monitor)
    at com.moi.DefaultImageProcessor$DownloadAndScaleCallable.call(
          DefaultImageProcessor.java:168)
    - waiting to lock <0x257aa6b8> 
     (com.moi.ImageUriMutexImpl$MutexImpl)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
        ThreadPoolExecutor.java:885)

I want to know how to read this stack dump. What signs should I look for in a deadlock ?

UPDATE I solved my problem! Basically I am doing a HttpClient 4.0 GET inside the synchronized block. The HttpClient was behaving badly & not returning & it held onto the locks. Via jstack, there were a couple of threads holding on to locks which caused the problem above. I understand now that it wasn't deadlocks so much but that my synchronized blocks were taking too long to return. Thanks. Merci.

Answer 1

Given you're noticing that the command "hangs", and you have identified two threads blocked on a mutex... I'd say you're reading the signs pretty well...

It really depends on what your service does. Look for performance issues and data consistency issues. For example, getting stuck and not responding, or even severe performance hits as the volume of requests go up are good signs you may have a problem. Additionally, Inconsistent data between multiple requests (again, depending on your service) may point to a problem as well.

Answer 2

From the little stack trace the threads are only waiting to acquire lock. In the trace look for the objects 0x257aa440 and 0x257aa6b8 and see who have locked those objects. Check if that thread is blocked.

In deadlock situation you would see a complete circle for blocked states. Also take the trace multiple times to confirm if the blocked state is momentary or a long wait.

Answer 3

These are two threads in contention for the same resource, so no this in iself isn't a problem. It might be half the problem though.

If these two threads are the only one's blocked, then you're not in deadlock. Deadlock (in it's simplest form) is where two threads each already have a lock on two different objects, but they each want a lock on the others one.

That said, just from what you have provided, no you're not in a deadlock. But if things are hanging, and starting to get backed up, then it's a good possibility, but it's impossible (or very hard atleast) to tell for sure from just a stack dump.

EDIT: Wait, they aren't locked on the same thing. But both threads are in the same function. I'd find it unlikely that this (alone) would be causing one, but it could part of a loop of threads causing a deadlock.

Answer 4

A (conventional) deadlock is easy to spot with jstack - it'll say there is a deadlock. (There can be cases where, for example, threads are waiting on each other but not on each others lock - EventQueue.invokeAndWait often causes this.)

What we have here are two threads blocks attempting to lock different objects (with identity hash codes 0x257aa440 and 0x257aa440). You will probably find other threads that actually do hold these locks (just use find in your text editor). It may be the case that the monitor was released very soon before and isn't held. In that case you're probably seeing highly contended locks.

Answer 5

Take a look at this question. This is an excerpt of the stacktrace that can be generated:

"Thread-1" prio=10 tid=0x0841ac00 nid=0x77d waiting for monitor entry [0xb42bf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at Deadlock$Friend.bowBack(Deadlock.java:16)
    - waiting to lock <0x8b80def8> (a Deadlock$Friend)
    at Deadlock$Friend.bow(Deadlock.java:13)
    - locked <0x8b80df08> (a Deadlock$Friend)
    at Deadlock$2.run(Deadlock.java:28)
    at java.lang.Thread.run(Thread.java:619)

"Thread-0" prio=10 tid=0x08419400 nid=0x77c waiting for monitor entry [0xb4310000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at Deadlock$Friend.bowBack(Deadlock.java:16)
    - waiting to lock <0x8b80df08> (a Deadlock$Friend)
    at Deadlock$Friend.bow(Deadlock.java:13)
    - locked <0x8b80def8> (a Deadlock$Friend)
    at Deadlock$1.run(Deadlock.java:25)
    at java.lang.Thread.run(Thread.java:619)



Found one Java-level deadlock:
=============================
"Thread-1":
  waiting to lock monitor 0x083f1464 (object 0x8b80def8, a Deadlock$Friend),
  which is held by "Thread-0"
"Thread-0":
  waiting to lock monitor 0x083efc90 (object 0x8b80df08, a Deadlock$Friend),
  which is held by "Thread-1"

Java stack information for the threads listed above:
===================================================
"Thread-1":
    at Deadlock$Friend.bowBack(Deadlock.java:16)
    - waiting to lock <0x8b80def8> (a Deadlock$Friend)
    at Deadlock$Friend.bow(Deadlock.java:13)
    - locked <0x8b80df08> (a Deadlock$Friend)
    at Deadlock$2.run(Deadlock.java:28)
    at java.lang.Thread.run(Thread.java:619)
"Thread-0":
    at Deadlock$Friend.bowBack(Deadlock.java:16)
    - waiting to lock <0x8b80df08> (a Deadlock$Friend)
    at Deadlock$Friend.bow(Deadlock.java:13)
    - locked <0x8b80def8> (a Deadlock$Friend)
    at Deadlock$1.run(Deadlock.java:25)
    at java.lang.Thread.run(Thread.java:619)

Found 1 deadlock.

So, when you have a deadlock at hand the VM can detect and show it.