How to do non-invasive profiling of a running ASP.NET application?

A few ideas:

Use custom Windows performance counters. They are very lightweight (1 to 2%), and you can use them not just to count, but also for timing measurements, to look at how often certain operations are slower than a threshold, etc.
If you're using IIS 7, you can enable Failed Request Tracing. To limit the perf impact, be careful to not enable it for too many pages. Those traces can provide lots of detail, and you can inject more info into them programmatically if you need to.
Use the Windows Event log to write custom details under exceptional conditions. Perf impact is minimal as long as you don't over-do it.
One cause of sawtooth memory behavior can be not calling Dispose() when you should (or wrapping IDisposable objects in using statements, which will call it for you); you might want to review your code to look for that.

In case it's helpful, you might also be interested in the performance tips from my book: Ultra-Fast ASP.NET.

Edit: you might also try using .NET Memory Profiler (free trial available) to attach to the running process. It's fairly invasive compared to counters, but if you need to capture a snapshot of the current state of memory to debug your problem, there may not be a better choice.

Looking into it, seems the path to take indeed.

Abel 2009-11-27 14:33:04

The most interesting thing is missing from that post: *making the memory dump*. How to?...

Abel 2009-11-27 14:46:54

Just did a memory dump, but it reset the app pool, stopped serving requests for a while (during the dump, but it took too long) and the actual dump is 200MB compared to 800MB for the process... Hmm... Is there something even less evasive available?

Abel 2009-11-27 15:15:07

Thanks for the advises. I'm aware of all of those and indeed, the pieces of the code we deemed the possible causes are not using `using` blocks around Disposable objects. But my gut feeling says, there's something else up here (ran smooth for a long time with 10x as much users, same code). Nothing odd in event log, no odd failed requests. Performance counters are on fulltime with us (see original question).

Abel 2009-11-27 14:50:16

From your question, I thought you were just using the existing Windows perf counters, not adding your own custom ones.... Threading / locking issues are another common cause of perf hiccups.

RickNZ 2009-11-27 14:53:04

Yes, perhaps my question wasn't very clear (sorry, tried to be brief, not my best asset). But I should do more with them (custom perf counters), but can't do a rebuild now, only monitor (important period for this shop). Threading and locks, or other type of wait loops should certainly be on our monitor, thanks.

Abel 2009-11-27 15:12:55

Sorry, I was trying to suggest that threading / locking issues combined with improper handling of IDisposables might explain why your code was working fine before and isn't any more. Custom instrumentation is often the least invasive (and sometimes the only) way to find problems like that (other than doing some detailed code walkthroughs).

RickNZ 2009-11-28 11:57:50

Another possibility is to use .NET Memory Profiler (see http://memprofiler.com/) -- I updated my answer with that info too.

RickNZ 2009-11-28 12:03:04

Not sure why I didn't find that earlier, but that's given *exactly* the information I was after. For some reason or another, I can't run it on production (simply too busy, tried it at night, but it totally chokes, so *non-invasive*, I take it, simply doesn't exist) but in the acceptance environment it seems to do just fine. Thanks for updating. Altogether this was most helpful and you did follow up on questions, so even though it's still invasive, it gets closer me where I want to be. Bottom line: there's no such thing as non-invasive profiling (yet).

Abel 2009-11-29 23:46:40

Excellent answer, esp. with the memprofiler in the edits.

Abel 2009-11-29 23:47:40

Not really. However, you can attach a debugger like windbg and basically get most of the same information -- it's a pretty invasive solution, though, particularly for a production environment.

RickNZ 2009-11-28 11:59:03

@RickNZ: It is invasive. Some folks make gadgets that glom onto a running process and take stackshots. I'll see if I can find the link.

Mike Dunlavey 2009-11-28 14:17:30

Thanks for your additional answer, Serge. I've worked with Yourkit quite a bit recently. However, what they claim on their homepage simply doesn't count. Even if you try to make the simplest memory snapshot with the server having only 5% of the normal amount of visitors, the server totally chokes and becomes unresponsive for more then an hour (then I killed it). Not a single request was handled completely during that time. Memory dumping as Tom suggested was less invasive. However, Yourkit has some excellent features which it seems to have copied 1:1 (incl imgs) from jetBrains dotTrace though.

Abel 2009-12-03 13:05:01

Btw, a major feature both dotTrace and Yourkit are missing is counting the occurrences of hitting a method in the CPU mode (knowing that one method takes 12% of total CPU is nice, but not knowing how often it's called makes this information less useful).

Abel 2009-12-03 13:07:08

ansaurus

tags:

views:

answers:

How to do non-invasive profiling of a running ASP.NET application?

related questions