How can I profile an ASP.NET application running on a production server?

Question

I have an ASP.NET application that is consistently using 75% - 100% of the CPU on a production server. How can I profile the application to figure out what part of the code is using up the most CPU? I have looked at a couple of different tools (Xte Profiler, EQATEC, dotTrace), but they all seem to want you to load and run the application within their tool. It seems to me that they want you to load up the application in their tool and run tests locally (not in production). I want to profile the application while it is running in production with people hitting it to see what is actually going on. Is this possible?

I am a newbie to application profiling so forgive me if I have missed something obvious or am not thinking about this correctly.

Thanks, Corey

Answer 1

My guess it has to do with long running database queries rather than the ASP.net application itself. In my experience 9 times out of 10 this is what I see and this takes the APPLICATION server down to a crawl as resources are consumed and the app has to wait for each query to finish to move on. Take a look at SQL profilier on the DB server and see if there are any queries that are taking a long time to execute.

It could be as simple as adding an index to a column or some other small minor optimizations. Once you know the query, you can then also go back to your code and tweak that section as well.

Answer 2

Your best bet is to profile your code on your own machine to identify where it is spending time.

Grab a ten day free trial of this:

http://www.jetbrains.com/profiler/

Here are some links to get you going:

http://blogs.msdn.com/b/alikl/archive/2009/03/06/why-my-asp-net-application-s-performance-is-less-than-optimal.aspx

http://msdn.microsoft.com/en-us/library/ms178643(v=VS.100).aspx

http://www.codeproject.com/KB/aspnet/10ASPNetPerformance.aspx

Answer 3

You should consider taking a memory dump on the production server while it's experiencing high CPU. Check out ADPlus and taking a hang dump on the asp.net process. This can then be analyzed with Windbg or other tools.

I just went through a similar experience where our production servers were experiencing excessive CPU load - a scenario we could not recreate locally or in test/staging environments. It had nothing to do with the database (database CPU was normal). Analyzing the dump file is what clued us in on what was causing the problem (excessive compilation of regex objects by some library we were using).

This answer would be incomplete without Tess' blog, so here's the link.

Answer 4

It's certainly possible to profile ASP.NET with the EQATEC Profiler. See:
http://stackoverflow.com/questions/378617/profiling-asp-net-websites-with-eqatec-profiler

EQATEC Profiler instruments your app in a separate step that enable the app itself to collect it's own profiling info, and the profiler then merely displays that timing data afterwards.

That means that you can run your instrumented ASP.NET app completely independent of the profiler itself. You could e.g. instrument your app, mail it to your test site in India, have them run it on their server for some days where it will generate timing reports all on it's own, and have them mail back those reports to you, which you can then view in the profiler. Pretty neat.

Note: To have the profiled app generate timing snapshots "on it's own" it must know when to generate them. By default this is when the method Application_End is called in an ASP.NET app. You can programmatically dump snapshots when it suits you by using the EQATEC Profiler API. See the user guide or check out this thread.