Web Based Stack Dump Tool for ASP.NET Using Mdbg?

Question

There is a great presentation by Dan Farino, Chief Systems Architect at MySpace.com, showcasing a web-based stack dump tool that catalogues all threads running in a given process (what they're doing, how long they've been executing, etc.)

Their techniques are also summarized on highscalability.com:

• PerfCollector.
  Centralized collection of performance data via UDP. More reliable than Windows and allows any client to connect and see stats.
• Web Based Stack Dump Tool.
  Can right-click on a problem server and get stack dump of the .Net managed threads. Used to have to RDC into system and attach a debugger and 1/2 later get an answer. Slow, nonscalable, and tedious. Not just a stack dump, gives a lot of context about what the thread is doing. Troubleshooting is easier because you can see 90 threads are blocked on a database so the database may be down.
• Web Base Heap Dump Tool.
  Dumps all memory allocations. Very useful for developers. Save hours of doing it by hand. • Profiler. Traces a request from start to finish and produces a report. See URL, methods, status, everything that will help you identify a slow request. Looks at lock contentions, are a lot of exceptions being thrown, anything that might be interesting. Very light weight. It's running on one box in every VIP (group of 100 servers) in production. Samples 1 thread every 10 seconds. Always tracing in background.

The question is: what tools are required to build a web-based stack dump tool for ASP.NET? For convenience, let's assume that an *.aspx hosted in the target AppDomain, able to output all managed call stacks in that process, is sufficient.

There are a few posts that cover the use of Mdbg (debugger for managed code written entirely in C#/IL that started shipping with CLR 2 SDK) and the mdbgcore assembly typically found in C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin:

• http://dotnetdebug.net/2005/11/09/exceptiondbg-v01-debug-your-exceptions/
• http://blogs.msdn.com/jmstall/archive/tags/MDbg/default.aspx
• http://blogs.msdn.com/vijaysk/archive/2009/11/04/asp-net-debugger-extension-for-iis-7.aspx

Would a solution simply reference this assembly to produce the desired output? What impact would a "list all managed call-stacks" operation have on the running process that's servicing production traffic?

Answer 1

I believe the profiling API of .Net are the way to go.

Look at the SlimTune project on Google Code to have a live sample, with sources, that you can check how to adapt and improve to work in a Asp.NET scenario.

Regards Massimo

Answer 2

With the profiling API of .Net you have to stop the server and it takes a lot CPU (but it gives you full control over all called methods).

I think the most “light way” solution is to doing this with MDbg, I put together a very small but useful little app called StackDump that does the following: 1) The debugger stops the application and generates a list of all CLR stacks running for the process. 2) The application is started again. This operation is a quick operation and can (maybe) be executed on a running production server with unchanged production code.

It just 80 lines of .Net code to manage this (I haven’t published the source code, but feel free to look at the code with reflector – and ignore all other namespaces then StackDump – because I linked in MDbgcore to get a easy to deploy application)