hints of impending failure

Question

I'm looking for indicators that I can programatically check that would hint at an impending failure of my application. I'm looking for general things like "the number of free IO threads is dangerously low", "memory available to the app pool is low", and "processor usage is high".

This is for C#/asp.net and IIS.

Examples of checking the value programatically would be nice, but not required. Any good ideas welcome.

Answer 1

It sounds like you're more interested in external causes of failure, not the app failing itself on its own merit. I think you're trying to engage in a sysyphean task here; instead of trying to guess at possible failure factors, focus on configuring your worker process to mitigate any of these issues. IIS worker processes are highly resilient - check out the ping, recyclying, etc features.

Answer 2

Maybe not programatic but keeping a tally of the number of hours supoort or overtime you have to perform to keep it from falling over may be a good indicator. :-)

Answer 3

You could programmatically access the values of some of the built-in system performance counters. If you have a need for even more specific details you could build your own counters to query as well.

Here's an article to get you started: How to calculate CPU usage programmatically

Answer 4

My thoughts:

Determine the performance inputs and outputs of your webapp.

• Inputs (number of users, what tasks users are doing, time of day, etc)
• Outputs (memory usuage, threads, lag)

Profile your application under real request data. Build a nice table of inputs and their associated outputs. For example:

4 users calling page1 -> costs 4   - 6   mb, 4 threads
5 users calling page1 -> costs 7   - 14  mb, 5 threads
2 users calling page2 -> costs 120 - 200 mb, 1 thread

Find outputs that often cause failure and find which inputs cause those outputs. Build a nice max-likelyhood model of failure.

When your inputs start approaching your failure outputs, impending failure is likely with some probability. Record when failure does and does not occur and feed this information back into your table. Your webapp will learn when it is about to fail.

UPDATED in response to comment:

Finding the outputs is the easiest part.

See SO questions How to get memory used in c#, How to get the cpu usage in C# and for the more general question What key performance monitors should I watch for asp.net application.

Key points from those questions:

• GC.GetTotalMemory - tells you how much is allocated by the Garbage Collector.

• Processor Object - tells you all sorts of interesting preformance stats on the cpu (cpu idle time, usauge, etc).

Answer 5

I actually had a job trying to figure this out a few years back. It turns out that this is a very, very difficult undertaking - these days, you can't even trust the OS, so running inside the .net sandbox will make it almost impossible to determine what's really going on.

Here are just a few simple situations you may want to take into consideration if you do decide to go ahead with this.

CPU - you may be able to determine the current CPU usage, but how do you tell if your particular instance has been bound to one CPU? How do you determine if you're running in a cluster, or what the other cores might be doing? You may be running inside a virtual machine with CPU or threads limited - those limits may be changed on the fly.

Memory - what if you're running inside a virtualized instance? What if suddenly an external change is make to increase or reduce the amount of memory available?