what can cause asp.net to stop responding

Question

we have an asp.net application, that makes heavy use of REST/json services through httpwebrequest and heavy use of the web cache. seemingly sporadically, the worker process simply stops responding. the browser sits expecting a response and never gets one. only an app pool restart seems to remedy the problem. there are no unhandled exceptions or any other anomalies that we have noticed. cpu and memory loads are low (<10% cpu util, >80% free memory)

the only possible hint that we have had is that we saw 'deadlock detected' messages in the event log at one point, but not consistently. additionally, we've eliminated what we think to be the only possible cause of such contention (executing multiple concurrent httpwebrequests via begin/end request).

any ideas?

win 03 server/data center edition (on amazon ec2) asp.net 3.5

Answer 1

When you have a non-responsive application with low cpu used, it's most likely a concurrency issue. You either have deadlocks which cause all your threads to stall, or your asynchronous threads take forever to finish and the ThreadPool ends up starving. Since you're running ASP.Net, you should check how the ThreadPool is doing. You can watch the performance counters or, better, attach to your application with windbg (you can get it from Debugging Tools for Windows http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx).

Launch windbg, attach to the asp.net process and then, in the command prompt:

.loadby mscorwks sos
!ThreadPool

Even better, download the sosex extension (http://www.stevestechspot.com/CommentView,guid,9fdcf4a4-6e09-4807-bc31-ac1adf836f6c.aspx) and check for deadlocks with

!dlk

check what are the threads blocking, switch to one of the threads (using the UI or the command line) and type

!CLRStack

to have a stack trace showing you which method is blocking your application.