Memory Leak with FileMonitorTarget / CacheDependency+DepFileInfo in ASP.NET

Question

In our ASP.NET web app we're experiencing a quite extensive memory leak which I am investigating right now. Using WinDbg I got down to the largest memory eaters in our app which are (ran !dumpheap -stat in the WinDbg console to get these):

MethodTable Addr   Count  Overall size Type
...
000007fee8306e10   212928     25551360 System.Web.UI.LiteralControl
000007feebf44748   705231     96776168 System.Object[]
000007fee838fd18  4394539    140625248 System.Web.Caching.CacheDependency+DepFileInfo
000007fee838e678  4394614    210941472 System.Web.FileMonitorTarget
000007feebf567b0    18259    267524784 System.Collections.Hashtable+bucket[]
00000000024897c0     1863    315249528      Free
000007feebf56cd0    14315    735545880 System.Byte[]
000007feebf4ec90  1293939   1532855608 System.String

For all I know a large number of String objects can be quite normal; still there's definitely room for improvement. But what really makes me itch is the count of System.Web.FileMonitorTarget objects: we have over 4 million instances on the heap (à 48 bytes)! Using two memory dumps and comparing them I've found out that these objects are not being cleaned up by the GC.

What I'm trying to find out is: where are these objects coming from? I've already tried ANTS Memory Profiler to get to the root of the evil but it leads nowhere near any of our own classes. I see the connection with System.Web.Caching.CacheDependency+DepFileInfo and thus the System.Web.Cache but we do not use file dependencies to invalidate our cache entries.

Also, there are 14315 instances of System.Byte[] making up for over 700 MB on the heap which stuns me - the only place where we use Byte[] is our image uploading component but we have only around 30 image uploads per day.

What might be the source of these Byte arrays and FileMonitorTarget objects? Any hints are very welcome!

Oliver

P.S. Someone asked pretty much the same question here but the only 'answer' there was very general.

Answer 1

There are a couple of things I would look into. You're right the strings are often used in great number. Still you have approx. 1.4 GB worth of strings on the heap. Does that sound right? If not I would look into that. If that is withing the expected range, just ignore it.

If you suspect FileMonitorTarget and/or Byte[] to be leaking, dump the instances using !dumpheap -mt XXX where XXX is the listed MethodTable for the types. You may want to use PSSCOR2 instead of SOS, as it makes this task a bit easier (the output from !dumpheap shows a delta column and you can limit the number of instances dumped).

The next thing to do is to start looking into what is keeping specific instances alive. The !gcroot command will tell you what roots a specific instance. Pick an instance at random and inspect the roots. If everything is as expected move on to the next. If you application is leaking instances of these types chances are that you will get an instance that should have been freed. Once you get the roots you need to figure out what part of the code is holding on to these. A common source is unsubscribed events, but there are other possible reasons why objects are kept alive.

Answer 2

Objects of type System.Web.Caching.CacheDependency+DepFileInfo are created automatically by ASP.NET to monitor file changes to your website. So even if you are not specifically using a FileDependency cache expiration, ASP.NET itself does.

If I run a dump field against some of these objects, I get a path to my controls/pages.

0:000> !df -field _filename 0d3f24ec 
Name: System.String
MethodTable: 79330b24
EEClass: 790ed65c
Size: 180(0xb4) bytes
GC Generation: 2
 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: C:\inetpub\wwwroot\Website\Application\Base\UserControl\Messages.ascx
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79332d70  4000096        4         System.Int32  1 instance       82 m_arrayLength
79332d70  4000097        8         System.Int32  1 instance       81 m_stringLength
79331804  4000098        c          System.Char  1 instance       44 m_firstChar
79330b24  4000099       10        System.String  0   shared   static Empty
    >> Domain:Value  000e0ba0:02581198 00109f28:02581198 <<
79331754  400009a       14        System.Char[]  0   shared   static WhitespaceChars
    >> Domain:Value  000e0ba0:025816f0 00109f28:02586410 <<

You can see this link describing a bit more detail: Understanding ASP.NET Dynamic Compilation

However, your case might still be different. Try running !GCRoot [obj_addr] and see what is holding onto those objects. In my case it is entirely IIS /.NET related objects.

That said, I still had a problem where millions of these cache objects were created, and I have no idea why. :| (this is the first time it happened to me, but I don't think it appeared or will disappear magically...)