Debugging ideas for a blocked Windows Message loop

Question

I have a longstanding C# .NET 3.5 application 'freeze' which I am at a loss with. There are two C# executables. One has a full UI, the other runs as a tray app. They both communicate via WCF to a third service app, also running in the tray.

Randomly the UI thread of main Winforms app will deadlock. Mysteriously if I quit the tray app the UI of this app will unlock.

Whenever I attach the debugger to either app I learn nothing useful. The UI thread is blocked in the frozen app on the Application.Run method. All other threads are either sleeping, or blocked on Invokes onto the UI thread.

Also mysteriously another running application like Photoshop will behave strangely whilst this deadlock is in place. Quitting the tray app sorts this too.

All I can deduce is that something is going wrong with the main Windows-level message pump, but I don't really understand how I can debug further into this. I've installed the framework source code and can see the deadlocked app is stuck in a while loop in :

Application.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop

but don't really understand enough to do anything with this information.

Does anyone have any advice at all on where to look further? I've been chasing this random deadlock bug for months.

Thanks, Nick

Answer 1

I think this could be a red herring since this one is in Visual Studio SDK so really your debugging freezes.

I have had to debug a few work related/work unrelated freezes and they are very very nasty and require meticulous instrumentation and code review. So be patient!

Here is a few pieces of advice from me:

1) You will see a few red herrings on the way so be careful not to get bugged down on them and confuse manifestations of the problem with the cause itself.

2) What is the timing of this freeze? How long does it take? A TCP connection time out usually takes 23 seconds while a database connection times out in 30, a command in 120 seconds (could be different on different settings) so the time it takes is a big clue. If it does not resolve by itself and you have to close one application to get rid of it, it is almost certainly a thread or database deadlock.

3) Use sysinternal's Process Explorer and Process Monitor to see what they are doing and at what point they freeze. The last sactivity could give you a hint not always.

4) I know it will take sometime but start writing tracing in your code so that you find the axact location of the issue and from then on, it usually takes a few hours to days to find the problem.

5) If you have more info, post another question and let me know.