Causes for web service memory leak

Question

We have a web service that uses up more and more private bytes until that application stops responding. The managed heap (mostly Gen2) will show some 200-250 MB, while private bytes shows over 1GB. What are possible causes of a memory leak outside of the managed heap?

I've already checked for the following:

1. Prolific dynamic assemblies (Xml serialization, regex, etc.)
2. Session state (turned off)
3. System.Policy.Evidence memory leak (SP1 installed)
4. Threading deadlock (no use of Join, only lock)
5. Use of SQLOLEDB (using SqlClient)

What other sources can I check for?

Answer 1

Make sure your app is complied in release mode. If you compile under debug mode, and deploy that, simply instantiating a class that has an event defined (event doesn't even need to be raised), will cause a small piece of memory to leak. Instantiating enough of these objects over a long enough period of time will cause all the memory to be used. I've seen web apps that would use up all the memory within a matter of hours, simply because a debug build was used. Compiling as a release build immediately and permanently fixed the problem.

Answer 2

Also look for:

• COM Assemblies being loaded
• DB Connections not being closed
• Cache & State (Session, Application)

Try forcing the Garbage Collector (GC) to run (write a page that does it when it loads) or try the instrumentation, but that's a bit hit and miss in my experience. Another thing would be to keep it running and see if it runs out of memory.

What could be happening is that there is plenty of memory and Windows does not signal your app to clean up. This causes the app to look like its using more and more memory because it can, when in fact the system can reclaim the memory when it needs. SQL Server and Exchange do this a lot. The idea is why cause a unnecessary cleanup when there are plenty of resources.

Rob

Answer 3

I would recommend you view snapshots of the stack at various times, and see what's using up the memory. If your application is using Java, then jmap works extremely well - you just give it the PID of the java process.

If using something else, try Lambda Probe (http://www.lambdaprobe.org/d/index.htm). It doesn't show as much detail, but will at least show you memory use.

I had a bad memory leak in my JDBC code that ended up being traced to a change in the JDBC specification a few years ago that I missed (with respect to closing statements and such). It took a combination of Lamdba Probe and then jmap to localize the problem enough to fix it.

Cheers,

-R

Answer 4

I'm sorry. I forgot to mention this is a .Net 3.5 web service, written in C#.

Also, several profilers (Ants, .Net Memory Profiler, etc.) all confirm that this leak is not in the managed heap.

Answer 5

Garbage collection does not run until a request for memory is denied due to lack of available memory. This can often make things look like a memory leak when one is not around.

Do you have any events and event handlers within the service? Services often have static variables, and if you are creating event handlers from the static instances, connected to a non-static instance object, the static will hold a reference to the instance forever, which will stop it from releasing.

Answer 6

Double check that trace is not enabled. I've seen instances of trace slowly consuming memory until the app reaches it's app pool limit.