We have some performance problems with one of our applications. I thought about using something like dotTrace to find out where the problems are, but dotTrace would probably degrade performance even more.
What's the best way to profile an application that's on a prod environment w/o affecting performance too much?
The general answer is "don't do it".
Other than that, you can gain a lot by using performance counters. If the built-in counters don't help, you can create your own.
Among other things, the performance counters may give you an idea of how to reproduce the performance problems through load testing.
The next idea is to narrow down the area you're interested in. There's no sense impacting performance for the entire application if it turns out to be your web service access that's slow.
Next, be sure to have instrumented your application, preferably by using configuration. The Enterprise Library Logging Application Block is great for that, as it allows you to add the logging to your application, but have it configured off. Then, you can configure what kind of information to log, and where to log it to.
This gives you choices about how expensive the logging should be, from logging to the event log to logging to an XML file. And you can decided this all at runtime.
Finally, you're not going to be able to use dotTrace or something else that requires restarting IIS an adding code to your running application. Not in production. The ideas above are for the purpose of not needing to do so.
You could use ants profiler
http://www.red-gate.com/products/ants_performance_profiler/index2.htm
They claim that "the overhead was hardly noticeable".
There is a 14 day free trial so you could give it a try.
Edit: I agree with John's comment, it will disrupt, require some down time, to get it started / stopped. Best to use it on a test environment to identify the bottle necks.
Did you load-test the application on a number of sessions that's anywhere near the actual load of the production environment?
The first thing that comes to mind is that your app is not scaling well under load or that your db is not scaling well with an increase in size (causing this way problems even with a very limited number of concurrent sessions) but it could be anything really.
My suggestion is to replicate the production environment and run proper load-testing then look at the data and it'll give you some clue.
you don't wanna play games with your production environment, but if you don't have it already you could use logging to keep track of the sequence and time spans of key-events and take it from there.
You can use ants profile as well as performance counter of the system. It will help you to determine whats the problem.
Here are some details about performance counter..
http://msdn.microsoft.com/en-us/library/fxk122b4.aspx