How can I find the cause of deadlock in multi-threaded c# program?

Question

Basically I have a c# server program (a console app not in IIS) which occasionally enters deadlock.

One thing that's strange is when I attach the debugger I see 3 threads waiting on a lock but no threads (using the threads window in visual studio) inside the lock! What's going on here .... I guess the visual studio debugger is lying.

But anyway .... what techniques or tools should I be using?

Thanks

Answer 1

I would start by sending trace output every time a thread is about to enter/leave a critical section as well as every time it successfully acquires a lock. Use the System.Diagnostics.Trace class.

Then you should be able to determine from your trace output which thread actually has the lock.

Typical trace code:

Trace.WriteLine("Acquiring lock - foo");
lock (foo)
{
    Trace.WriteLine("Acquired lock - foo");
    // Do some stuff
    Trace.WriteLine("Releasing lock - foo");
}
Trace.WriteLine("Released lock - foo");

Depending on the way your program is structured, this might not be useful unless you include thread information in the trace output, for example:

Trace.WriteLine(string.Format("Thread {0} - Acquiring lock - foo",
    Thread.CurrentThread.ManagedThreadId);

Once you've figured out which thread has the lock, you can go into the debugger and see which lock it's waiting for, then use the same trace output to figure out who has the other lock. In most cases there will be two threads participating in a deadlock and this will let you find both of them.

Answer 2

Hi there.

You can also use WinDbg with the SOS.dll to find out where a deadlock resides. Have a look at this article. The SOS.dll command in question is !syncblk - it scans .NET threads looking for information about potential deadlocks.

Cheers. Jas.

Answer 3

You probably have a race condition with your threads. One of your threads is not giving up a lock and it's not using it or allowing anyone else to use it. It also happens when a thread has the lock and is killed before giving it up. You might want to check out any of the programming solutions for deadlock, such as Sleeping Barber, Lamport's Bakery, Dekker's Algorithm, Peterson's Algorithm or Dining Philosophers.

• http://en.wikipedia.org/wiki/Lamport's_bakery_algorithm
• http://en.wikipedia.org/wiki/Dining_philosophers_problem
• http://en.wikipedia.org/wiki/Peterson's_algorithm
• http://en.wikipedia.org/wiki/Dekker's_algorithm
• http://en.wikipedia.org/wiki/Sleeping_barber_problem

I don't know how you're managing your threads, so I can't tell you which of the algorithms are the right solutions, but they are the most common solutions for concurrency and deadlock management inside of Operating Systems, so they should handle your problem.

Answer 4

There is always WinDbg, but the learning curve can be a bit daunting.

But see Brians excellent answer on this question: http://stackoverflow.com/questions/508398/detecting-deadlocks-in-a-c-application/508478#508478

Answer 5

In a word, Chess, a program from Microsoft for helping solve exactly these kinds of problems.

From their site:

CHESS repeatedly runs a concurrent test ensuring that every run takes a different interleaving. If an interleaving results in an error, CHESS can reproduce the interleaving for improved debugging. CHESS is available for both managed and native programs.