Diagnosing runaway CPU in a .Net production application

Question

Does anyone know of a tool that can help me figure out why we are seeing runaway CPU in a managed app?

What I am not looking for:

1. Process explorer, it has this awesome feature that lets you see CPU per thread, but you do not get managed stack traces. Also, it requires a fairly proficient user.

2. Windbg + SOS, it could probably be used to figure out what is going on, by grabbing a bunch of dumps. But it is non-trivial to automate and a bit to heavy for this.

3. Fully fledged profiler (like dottrace or redgate), licensing is complex and the tool is an overkill which requires a reasonably heavy install.

What I am looking for:

1. A simple exe (with no installer) I can send to a customer. After they run it for 10 minutes, it generates a file that they send to me. The file contains details on the threads that consumed the most CPU and their stack traces during that time.

Technically I know that a tool like this can be created (using ICorDebug), but do not want to invest any time if such a tool already exists.

So, anyone know of anything like this?

Answer 1

A profiler is probably the correct answer here.

If you don't want a "fully fledged profiler" like DotTrace, you might want to try SlimTune. It works fairly well, and is completely free (and open source).

Answer 2

I've had luck with the Red Gate Ants profiler. However it does require installation. I'm pretty sure they don't have a remote option.

Answer 3

I know that you specifically said that you didn't want to take complicate dumps and use WinDbg + Sos to analyze those.

However, that may not be necessary. I would suggest using WinDbg anyway, but instead of using dumps, just attach to the process when you see the runaway thread(s). Then all you need to do is run the !runaway command. That will give you the total running time for each thread. The runaway threads will be at the top of the list. Now all you have to do is run the !clrstack for the top thread (or threads as it may be).

E.g. if thread 4 is your primary suspect, do a ~4e!clrstack to get the managed stack for that thread. That should tell you what the runaway thread is doing.

I'll agree that WinDbg is not the easiest tool to use for many things, but this may actually turn out to be pretty simple, so I hope that you'll forgive me for posting something you didn't really what.

If WinDbg is still out of the question, feel free to comment.

Answer 4

It does sound like you need a real profiler, but I thought I'd just throw this out there: PerfMon. It comes with windows, you can setup a perfmon profile that you can send to the user, they can capture and send you the log.

Here's a couple links I've kept around every time I need a perfmon refresher: TechNet magazine from 2008 and a post from the Advanced .NET Debugging blog.

Answer 5

If you have managed code in lieu of a profiler worth using I've found that throwing a log message into your code is damn useful for spotting infinite loops and general multi thread progressions.

i.e

step 1 msg
step 2 msg

thread now 100% and no step 3 msg = bug.

Answer 6

I think you should look at memory and disk usage as well. If a machine runs out of memory and needs to start using virtual memory (on the disk drive), you'll see a spike in CPU and disk activity. In such conditions what looks like a CPU bottleneck is actually a memory bottleneck.

Answer 7

The basic solution

1. Grab managed stack traces of each managed thread.
2. Grab basic thread statistics for each managed thread (user mode and kernel time)
3. Wait a bit
4. Repeat (1-3)
5. Analyze the results and find the threads consuming the largest amount of cpu usage, present the stack traces of those threads to the user.

Managed Vs. Unmanged Stack Traces

There is a big difference between managed and unmanged stack traces. Managed stack traces contain information about actual .Net calls whereas unmanaged ones contain a list of unmanaged function pointers. Since .Net is jitted the addressed of the unmanaged function pointers are of little use when diagnosing a problem with managed applications.

How do you get an unmanaged stack trace for an arbitrary .Net process?

There are two ways you could get managed stack traces for an managed application.

• Use CLR profiling (aka. ICorProfiler API)
• Use CLR Debugging (aka. ICorDebug API)

What is better in production?

The CLR Debugging APIs have a very important advantage over the profiling ones, they allow you to attach to a running process. This can be critical when diagnosing performance issues in production. Quite often runaway CPU pops up after days of application use due to some unexpected branch of code executing. At that point of time restarting the app (in order to profile it) is not an option.

cpu-analyzer.exe

So, I wrote a little tool that has no-installer and performs the basic solution above using ICorDebug. Its based off the mdbg source which is all merged into a single exe.

It takes a configurable (default is 10) number of stack traces for all managed threads, at a configurable interval (default is 1000ms).

Here is a sample output:

C:\>cpu-analyzer.exe evilapp
------------------------------------
4948
Kernel Time: 0 User Time: 89856576
EvilApp.Program.MisterEvil
EvilApp.Program.b__0
System.Threading.ExecutionContext.Run
System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal
System.Threading._ThreadPoolWaitCallback.PerformWaitCallback

... more data omitted ...

Feel free to give the tool a shot. It can be downloaded from my blog.

EDIT

Here is a thread showing how I use cpu-analyzer to diagnose such an issue in a production app.

Answer 8

The worse a problem is, the easier it is to find by this technique.

There is a tool you can get, called Stackshot, that might help in your case. Look here and here.

Answer 9

Use SysInternals ProcDump to get a mini-dump and windbg+sos to analyse it.

The ProcDump utility is available here: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

Just send the exe to the user and tell him.her to run (for example):

ProcDump MyProgram.exe -c 90 -s 10

This will dump the process if it's consuming over 90% CPU for more then 10 seconds

Answer 10

Use the Managed debugger. Helped me before. Just a few files needed. You could probably just see what is happening (perhaps exception handling stuck in a loop).