Garbage Collection does not reduce current memory usage - in release mode. Why?

Question

I built a quick program that needed to loop through an enormous log file (a couple of million records) and find various bits and pieces from inside. Because the volume of data was so huge, I have been curious to watch my Windows Task Manager performance tab and see how much CPU and memory is being used.

After the program successfully gets my results, the CPU usage goes right down, obviously. But after watching my memory usage slowly go up to several gigabytes during execution, it stays the same.

I have tried to call GC.Collect() at the end of my function, tried setting things to null, tried to run the program in Release mode (I heard that GC.Collect() might not work as I want it to in debug mode).

The memory usage goes right down if I close the program, but I can't figure out why I can't clean my app up during the apps lifetime. At the end of the day this is a throwaway app, but I'm just curious to know what I'm missing i.e. what is holding on to all this memory.

Thoughts?

Answer 1

Calling GC.Collect() is only a request to collect garbage, it's not an order, so the CLR may choose to ignore the request if it sees fit.

For example, if there's a lot of garbage that would cause a noticable delay whilst collecting, but there's still free memory to service subsequent allocations, then the GC may opt to ignore your request.

Answer 2

The garbage collector is not guaranteed to run when you call Collect(). It simply flags the object for collection. The next time the GC runs it will "collect" the flagged object.

There is no way in .NET to force the GC to collect at a specific point in time - if you need this functionality you'll need to go to native code.

Answer 3

Do any of the objects used in your loop implement IDisposable?

Answer 4

The Collect method does not guarantee that all inaccessible memory is reclaimed.

Answer 5

Don't forget that the CLR is ultimately using the underlying memory manager in the Windows kernel, which may have its own policies about when to release memory. For example, if the memory you're using comes from a pool, it may just be returned to the pool rather than deallocated when you free it at the user level. So it's entirely possible that there could be (and probably will be) a discrepancy between how much memory your app is holding and how much memory is allocated to the process.

Answer 6

You should look at the performance counters for the sizes of the CLR heaps, not the total memory allocated to the process shown in the task manager. Windows won't reclaim the memory allocated to the process unless the system as a whole is starved for memory, even if the process doesn't use the memory for anything.

Answer 7

It's normal for many memory allocators (with or w/o garbage collection) to "hoard" the memory they've freed, rather than give it back to the OS, because in many situations the memory just freed will be requested again and it's faster to keep it in the process rather than keep giving it back and asking the OS for it again and again (giving it back &c also requires extra effort due to page alignment issues and the like).

In a one-shot app, as you mention, that's not a problem. When I have a long-running app that has what I know will be a transient requirement for a lot of memory, one approach I use is to spawn a sub-process to perform the memory-hungry stuff, since when that process ends the OS will claw back the memory. It's easier on Unix systems (just fork) but not too bad on Windows, either, when warranted.

Answer 8

It is possible that your objects are still rooted somehow. You could try using a memory profiler to see if this is the case. I usually recommend SciTech's .NET Memory Profiler to do this but Red-Gate also has a decent memory profiler. Both SciTech and Red-Gate have trial versions available. It is also possible using WinDBG with SOS.

There is a somewhat out of date list of all profilers here.

Answer 9

to force GC.Collect to do collection, call it like this GC.Collect(2); this means you are asking the GC to do a full blocking GC for the whole heap, 2 means Generation 2 of your heap.