.NET OutOfMemory exception

Question

We have a .NET application running in production for over a year. It is a background service and the way it is written is that if it crashes for whatever reason it will be automatically restarted. Till recently it did not give us any headaches despite a heavy load it handles daily, but now every once in a blue moon it crashes with OutOfMemorey exception. After restart it picks up whereever it crashed and does whatever it is supposed to do, so the customer does not care, but I would like to understand what causes the crash.

Therefore the question: Is there a way to examine the crash WITHOUT redeploying the app? What I would like to do is to generate a crash dump or smething and then manually go through the dump in an attempt to figure out what objects ate all my memory. What tools would you suggest I use to make this task easier?

Answer 1

You may find the following article from MSDN helpful:

Production Debugging for .NET Framework Applications

http://msdn.microsoft.com/en-us/library/ms954591.aspx

Although it discusses ASP.NET applications in detail, some of the techniques describes are just as valid when performing memory analysis on services or stand-alone applications.

You may also want to use Perfmon to connect to the process and see if you are getting a lot of growth in the Gen 1 or Gen 2 heaps - this is usually indicative of objects that are surviving too many garbage collections and creating memory pressure in your application. You may also want to look at the size of the Private Bytes metric - growth there indicates that a lot of unmanaged memory is getting allocated for your process.

Perfmon analysis is useful because it can show you memory usage overtime, and help pinpoint if particular types of processing or transactions are responsible for memory consumption.

If you can reproduce the problem in a test environment, there is an excellent memory profiler from the folks at Red Gate Software that can help pinpoint such problems in a fraction of the time it would take digging through memory dumps.

There are some tools that can run in the background (such as DebugDiag) that allow cpaturing memory dumps periodically, or when certain events occur. You can read about them here:

http://blogs.msdn.com/sukeshak/pages/ddintro.aspx

http://blogs.msdn.com/tess/archive/2009/01/23/net-hang-analyzing-debug-diag-output.aspx