+1  A: 

There are two places you should focus.

First, in your test environment, you should be using a code profiling tool (like the kind built into VS) to see what code paths are actually being executed. This reveals by far the most details into your app, but it can be difficult and time consuming to set up a good simulation of your production environment and load.

Second, use custom performance counters in your production code. We create our own performance counters, and install them with a setup utility. This adds some complexity, but you really can't beat having them for insight into what your web application is doing.

The act of updating a performance counter is minimal. It's the reading of the performance monitor which can add some overhead (counter-intuitive, I know).

The version of the Enterprise Library Library we're using (3.1) does not have any direct support for custom performance counters, although it does install a number of counters for itself. I don't think these are the ones you want.

Alan McBee
We have 2 performance counters but it is a bit like the oil light on a car dashboard. If it lights up your oil pressure has dropped already and it is probably to late to do anything about it. I guess the correct counters to use is very much dependant on your application and what it is doing. Thanks for your advice. Do you use installutil to install the performance counters?
Rihan Meij
We would use installutil if we were installing the assemblies that have the performance counters into the GAC; it seems the VS2005 setup projects don't like the combination of installing to GAC + running custom actions to install perf counters. But we don't, so we just use custom actions. Also, use perf counters to regionalize the problem, but then use conditional logging (EntLib log categories) to pinpoint. But really? Profile. Ounce of prevention is worth a pound of cure.
Alan McBee
Also, add more performance counters. We have at least eight on our smallest projects. Measure elapsed time, measure counts, measure rates.
Alan McBee
+1  A: 

Take a look at Design for Operations. This includes tools that will help you instrument your application. These tools also help you create a health model for your application that can be used by MOM for health monitoring.

John Saunders