Why does IHttpAsyncHandler leak memory under load?

Question

I have noticed that the .NET IHttpAsyncHandler (and the IHttpHandler, to a lesser degree) leak memory when subjected to concurrent web requests.

In my tests, the Visual Studio web server (Cassini) jumps from 6MB memory to over 100MB, and once the test is finished, none of it is reclaimed.

The problem can be reproduced easily. Create a new solution (LeakyHandler) with two projects:

1. An ASP.NET web application (LeakyHandler.WebApp)
2. A Console application (LeakyHandler.ConsoleApp)

In LeakyHandler.WebApp:

1. Create a class called TestHandler that implements IHttpAsyncHandler.
2. In the request processing, do a brief Sleep and end the response.
3. Add the HTTP handler to Web.config as test.ashx.

In LeakyHandler.ConsoleApp:

1. Generate a large number of HttpWebRequests to test.ashx and execute them asynchronously.

As the number of HttpWebRequests (sampleSize) is increased, the memory leak is made more and more apparent.

LeakyHandler.WebApp > TestHandler.cs

namespace LeakyHandler.WebApp
{
    public class TestHandler : IHttpAsyncHandler
    {
        #region IHttpAsyncHandler Members

        private ProcessRequestDelegate Delegate { get; set; }
        public delegate void ProcessRequestDelegate(HttpContext context);

        public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback cb, object extraData)
        {
            Delegate = ProcessRequest;
            return Delegate.BeginInvoke(context, cb, extraData);
        }

        public void EndProcessRequest(IAsyncResult result)
        {
            Delegate.EndInvoke(result);
        }

        #endregion

        #region IHttpHandler Members

        public bool IsReusable
        {
            get { return true; }
        }

        public void ProcessRequest(HttpContext context)
        {
            Thread.Sleep(10);
            context.Response.End();
        }

        #endregion
    }
}

LeakyHandler.WebApp > Web.config

<?xml version="1.0"?>

<configuration>
    <system.web>
        <compilation debug="false" />
        <httpHandlers>
            <add verb="POST" path="test.ashx" type="LeakyHandler.WebApp.TestHandler" />
        </httpHandlers>
    </system.web>
</configuration>

LeakyHandler.ConsoleApp > Program.cs

namespace LeakyHandler.ConsoleApp
{
    class Program
    {
        private static int sampleSize = 10000;
        private static int startedCount = 0;
        private static int completedCount = 0;

        static void Main(string[] args)
        {
            Console.WriteLine("Press any key to start.");
            Console.ReadKey();

            string url = "http://localhost:3000/test.ashx";
            for (int i = 0; i < sampleSize; i++)
            {
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
                request.Method = "POST";
                request.BeginGetResponse(GetResponseCallback, request);

                Console.WriteLine("S: " + Interlocked.Increment(ref startedCount));
            }

            Console.ReadKey();
        }

        static void GetResponseCallback(IAsyncResult result)
        {
            HttpWebRequest request = (HttpWebRequest)result.AsyncState;
            HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result);
            try
            {
                using (Stream stream = response.GetResponseStream())
                {
                    using (StreamReader streamReader = new StreamReader(stream))
                    {
                        streamReader.ReadToEnd();
                        System.Console.WriteLine("C: " + Interlocked.Increment(ref completedCount));
                    }
                }
                response.Close();
            }
            catch (Exception ex)
            {
                System.Console.WriteLine("Error processing response: " + ex.Message);
            }
        }
    }
}

Debugging Update

I used WinDbg to look into the dump files, and a few suspicious types are being held in memory and never released. Each time I run a test with a sample size of 10,000, I end up with 10,000 more of these objects being held in memory.

• System.Runtime.Remoting.ServerIdentity
• System.Runtime.Remoting.ObjRef
• Microsoft.VisualStudio.WebHost.Connection
• System.Runtime.Remoting.Messaging.StackBuilderSink
• System.Runtime.Remoting.ChannelInfo
• System.Runtime.Remoting.Messaging.ServerObjectTerminatorSink

These objects lie in the Generation 2 heap and are not collected, even after a forced full garbage collection.

Important Note

The problem exists even when forcing sequential requests and even without the Thread.Sleep(10) in ProcessRequest, it's just a lot more subtle. The example exacerbates the problem by making it more readily apparent, but the fundamentals are the same.

Answer 1

It will be very likely a problem in your code.

The first thing I would check is if you detach all the event-handlers in your code. Every += should be mirrored by a -= of an event handler. This is how most .Net application create a memory leak.

Answer 2

There is no problem with your code, and there is no memory leak. Try running your console application a couple times in a row (or increase your sample size, but beware that eventually your http requests will be rejected if you have too many concurrent requests sleeping).

With your code, I found that when the web server memory usage reached about 130MB, there was enough pressure that garbage collection kicked in and reduced it to about 60MB.

Maybe it's not the behavior you expect, but the runtime has decided it is more important to quickly respond to your rapid incoming requests than to spend time with the GC.

Answer 3

The memory you are measuring may be allocated but unused by the CLR. To check try calling:

GC.Collect();
context.Response.Write(GC.GetTotalMemory(true));

in ProcessRequest(). Have your console app report that response from the server to see how much memory is really used by live objects. If this number remains fairly stable then the CLR is just neglecting to do a GC because it thinks it has enough RAM available as it is. If that number is increasing steadily then you really do have a memory leak, which you could troublshoot with WinDbg and SOS.dll or other (commercial) tools.

Edit: OK, looks like you have a real memory leak then. The next step is to figure out what's holding onto those objects. You can use the SOS !gcroot command for that. There is a good explanation for .NET 2.0 here, but if you can reproduce this on .NET 4.0 then its SOS.dll has much better tools to help you - see http://blogs.msdn.com/tess/archive/2010/03/01/new-commands-in-sos-for-net-4-0-part-1.aspx

Answer 4

I've had a look at your code (and run it) and I don't believe the increasing memory you are seeing is actually a memory leak.

The problem you've got is that your calling code (the console app) is essentially running in a tight loop.

However, your handler has to process each request, and is additionally being "nobbled" by the Thread.Sleep(10). The practical upshot of this is that your handler can't keep up with the requests coming in, so its "working set" grows and grows as more requests queue up, waiting to be processed.

I took your code and added an AutoResetEvent to the console app, doing a

.WaitOne() after request.BeginGetResponse(GetResponseCallback, request);

and a

.Set() after streamReader.ReadToEnd();

This has the effect of synchronising the calls so the next call can't be made until the first call has called back (and completed). The behaviour you are seeing goes away.

In summary, I think this is purely an runaway situation and not actually a memory leak at all.

Note: I monitored the memory with the following, in the GetResponseCallback method:

 GC.Collect();
 GC.WaitForPendingFinalizers();
 Console.WriteLine(GC.GetTotalMemory(true));

[Edit in response to comment from Anton] I'm not suggesting there is no problem here at all. If your usage scenario is such that this hammering of the handler is a real usage scenario, then clearly you have an issue. My point is that it is not a memeory leak issue, but a capacity issue. The way to approach solving this would be, maybe, to write a handler that could run faster, or to scale out to multiple servers, etc, etc.

A leak is when resources are held onto after they are finished with, increasing the size of the working set. These resources have not been "finished with", they are in a queue and waiting to be serviced. Once they are complete I believe they are being released correctly.

[Edit in response to Anton's further comments] OK - I've uncovered something! I think this is a Cassini issue that doesn't occur under IIS. Are you running your handler under Cassini (The Visual Studio Development Web Server)?

I too see these leaky System.Runtime.Remoting namespace instances when I am running under Cassini only. I do not see them if I set the handler up to run under IIS. Can you confirm if this is the case for you?

This reminds me of some other remoting/Cassini issue I've seen. IIRC, having an instance of something like an IPrincipal that needs to exist in the BeginRequest of a module, and also at the end of the module lifecycle, needs to derive from MarshalByRefObject in Cassini but not IIS. For some reason it seems Cassini is doing some remoting internally that IIS isn't.