Current Classes Loaded Constantly Increasing - Memory Leak

Question

I've got a service that has a very slow memory leak. If I analyze the .NET CLR Loading counters, I see that the Current Classes Loaded counter is constantly increasing and matches Total Classes Loaded counter at all times. This gives me the impression that the memory leak is related to resources not being freed (This is just a guess).

The service creates new appDomains each time it performs a task (plug-in architecture).

I need to figure out the class names so I can narrow down the cause of the leak. I'm not very proficient with WinDbg, but I was wondering if anyone could walk me through the steps to enumerate these Loaded classes.

I do have the source code so I can obtain the symbol files if necessary. Thanks in advance for any help!

Answer 1

I would suggest using a proper memory profiler - like ".NET Memory Profiler" - http://memprofiler.com/ You can certainly try it out on evaluation to see if it's the kind of tool which will help.

That will allow you to see all the live objects much more easily than hardcore WinDBG/SoS stuff.

Answer 2

EDIT: After reading some of the other posts, these should be taken into consideration after a better profiler is used and after the appDomain issue is sorted out.

You may want to add performance counters into your service so at least the objects you create can be tracked. That will help you determine if the Classes.Loaded are yours, or CLR classes.

Also adding some debug logging for when you explicitly create and destroy your objects may be helpful.

As a last resort, you can try putting a GC.Collect() in there to see if calling it corrects your problem. Its not the fix, but testing it will let you know it is an option.

Answer 3

Is this .net 2.0 or higher? If so, you may not be unloading the AppDomain (as Jon Skeet says in the comment).

If it's 1.1 or lower, there is a bug in the AppDomain unload functionality. I.e. it doesn't free up the memory or release resources when an AppDomain is "unloaded".

(This was fixed as of .net 2.0)

Answer 4

As said by the other answer, AppDomain.Unload() should be used.

However, you need to be careful you do not load assemblies in multiple places, especially the primary appdomain.

Look at the MSDN docs for AppDomain.Load(AssemblyName) for an explanation of how the above happens.

Also in the same line, are you sure you are using proper remotable classes? If not, the above will surely happen.

Answer 5

The AppDomains are unloaded, but the response from leppie, makes me wonder if the plugin-assemblies are being loaded into both the Primary AppDomain and the secondary AppDomain. When I look at the performance counters, the current AppDomain count does not constantly increase.

The application is supposed to create an Secondary appDomain and then load a separate plugin assembly. Maybe some code would help:

Creating the secondary AppDomain from the primary appDomain:

AppDomainSetup ads = new AppDomainSetup();
ads.ApplicationName = "RemoteAgentLib";
ads.ApplicationBase = AppDomain.CurrentDomain.BaseDirectory;
ads.PrivateBinPath = AppDomain.CurrentDomain.BaseDirectory;
ads.ShadowCopyDirectories = AppDomain.CurrentDomain.BaseDirectory;
ads.ShadowCopyFiles = "true";

m_domain = AppDomain.CreateDomain("RemoteTaskRunner", null, ads);

Use the RemoteTaskRunner to load the plugin in the secondary appDomain:

RemoteTaskRunner taskRunner = m_domain.CreateInstanceAndUnwrap(
                    Assembly.GetExecutingAssembly().FullName,
                    typeof (RemoteTaskRunner).FullName) as RemoteTaskRunner;
taskRunner.LoadTask(taskInfo.Assembly, taskInfo.Type);

Use the RemoteTaskRunner to Execute the task in the secondary appDomain:

[Serializable]
internal class RemoteTaskRunner : MarshalByRefObject
{
    private ITask m_task;

    public RemoteTaskRunner()
    {
    }

    internal void LoadTask(string assembly, string type)
    {
     // This assembly should load in the secondary appDomain.
     Assembly taskAssembly = AppDomain.CurrentDomain.Load(assembly);
     m_task = taskAssembly.CreateInstance(type) as ITask;
    }

    internal void RunTask(string taskConfig)
    {
     // This method should run in the secondary appDomain.
     m_task.RunTask(taskConfig, m_logger);
    }
...
...

To execute the plugin task, the following line of code is used in the Primary appDomain:

taskRunner.RunTask(taskInfo.TaskConfig);

After the task finishes, the appDomain is unloaded:

AppDomain.Unload(m_domain);

Answer 6

I just read this on Suzanne Cook's blog.

http://blogs.msdn.com/suzcook/archive/2003/06/12/57169.aspx

Be sure to not pass any Type/Assembly/etc. instances (besides your MarshalByRefObject type) back to the original appdomain. If you do, it will cause those assemblies to be loaded into the original appdomain. If the appdomain settings are different between the two appdomains, those assemblies may not be loadable there. Plus, even if they are successfully loaded, the assemblies will remain loaded and locked after the target appdomain is unloaded, even if the original appdomain never uses them.

When she says any Type/Assembly/etc. What does she may ANY type? The reason I ask, is because my MarshalByRefObject (RemoteTaskRunner) does return a DateTime object back after the task runs. Could this cause the plugin assembly to get loaded into my primary appDomain (and ultimately cause the memory leak)?

Answer 7

You can always check what assemblies are loaded in your AppDomain:

foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies())
{
      Console.WriteLine(assembly.FullName);
}

So if you accidentally load the assemblies in the wrong domain it wont be hard too see.

Edit:

if you want to use WinDgb SOS, here are the supported commands. You're most likely interested in: DumpDomain, DumpClass, DumpAssembly, EEHeap ...

Answer 8

I believe that the issue was actually caused by a series of un-disposed FileSystemWatcher instances that were nested way down inside the RemoteTaskRunner MBRO. I'm still not sure I have resolved the memory leak entirely, but I can definitely tell a difference.

It seems like this isn't the first time FileSystemWatchers have caused me issues. :)

Thanks everyone (especially leppie) for helping me with this!

Answer 9

I always throw this out there, whenever somebody reports a memory leak, because it kept me busy for a couple weeks. Don't run your app in debug mode. If you run your app in debug mode in .Net 2.0+ (didn't happen in .Net 1.1), and you instantiate a class that contains an event, even if you don't raise the event, it will hold only a small piece of memory. This can greatly effect long running apps, such as services and web apps, because over time the small amount of memory eaten up after instantiating the objects can add up to quite a lot.