tags:

views:

2362

answers:

8

Hi, I've create a c# application which uses up 150mb of memory (private bytes), mainly due to a big dictionary:

Dictionary<string, int> Txns = new Dictionary<string, int>();

I was wondering how to free this memory up. I've tried this:

Txns = null;
GC.Collect();

But it doesn't seem to make much of a dent in my private bytes - they drop from say 155mb to 145mb. Any clues?

Thanks

-edit-

Okay i'm having more luck with this code (it gets the private bytes down to 50mb), but why?

Txns.Clear(); // <- makes all the difference
Txns = null;
GC.Collect();

-edit-

Okay for everyone who says 'dont use gc.collect', fair enough (i'm not going to debate that, other than saying you can see my C background coming through), but it doesn't really answer my question: Why does the garbage collector only free the memory if i clear the transaction list first? Shouldn't it free the memory anyway, since the dictionary has been dereferenced?

+2  A: 

Not sure from memory if Dictionary has a Dispose() or not, but it's bound to have a Clear(). Call either of these before setting any references to null.

Then, simply let the garbage collector do its work. It is almost never a good idea to call GC.Collect() explicitly yourself and it might not even do what you want/need/expect and end up costing you performance. Static Code Analysis (=FxCop) doesn't warn you with Reliability rule CA2001 about this for nothing, you know? Simply do not do this unless you really know what you're doing. And even then don't do it. ;-)

Are you sure the dictionary is that huge? Isn't it just 10 Mb of memory and the rest is otherwise taken by your application? Question that might help you: Have you used a profiler yet to see where memory is actually consumed...?

peSHIr
The dictionary *is* that big. When i ran the code with the .Clear change, memory usage goes from 160mb to 50mb. So i assume the dictionary is 110mb.
Chris
+1  A: 

It's not usually a good idea trying to force the GC. Do you really need to have the whole dictionary in memory?

ktulur
Good question. Could be a design issue.
peSHIr
Yes i need it, its for a big reconciliation (compare rows from one table to the other). 150mb is peanuts.
Chris
150 mb is hardly peanuts. If you really need it, that's fine and dandy, but a good majority of the time, statements like that imply some sort of design issue or ignorance regarding a better way to do it. I mean that as no insult, just an observation.
Chris
+3  A: 

if u call GC.Collect() it start to do its job , but it returns immediately, it doesnt block, so you dont see its affect, if you just call GC.WaitForPendingFinalizers() after that it will block your app until GC.Collect() finish its job

Adinochestva
Yikes! Can't even begin to think about the levels of badness of starting a thread that periodially (and rather often: it tries twice a second!) calls GC.Collect and then waits for all pending finalizers. Are you just totally not interested in scaling applications? That's just trying to hide a bad symptom, badly! Guess I'll just have to -1 here...
peSHIr
@peSHIr: agreed, though to be fair, it does explicitly answer the question. Whether is a good idea or not is a different question.
andy
@andy: begrudgingly agreed. Should I -1 the question instead then...? Also seems unnecessarily harsh and useless. Oh well.
peSHIr
I'm not sure that is entirely correct. A garbage collection will (may?) wake up the finalizer thread, which will then run finalizers as needed. Thus waiting for the finalizer is not really waiting for the garbage collection to finish (as GC happens on a different thread)
Brian Rasmussen
Hi, Thanks for this answer but it doesn't seem to make a difference, as i tried this and the private bytes didn't go down: //Txns.Clear(); Txns = null; for (int x = 0; x < 20; x++) { GC.Collect(); GC.WaitForPendingFinalizers(); }Now keep in mind everyone this is proof of concept code! Its not going into production. But the only way i can get it to free the memory is calling the dictionary's Clear method, which i _really shouldn't_ have to call, since i'm dereferencing the whole dictionary.
Chris
Peshlr - who mentioned starting threads and running the GC twice a second? Is that an earlier revision of the above answer?
Chris
+2  A: 

Most probably you have a hidden reference to the dictionary somewhere else. Thus the dictionary is not collected, but if you Clear() it, the contents get collected.

As others have already noted, forcing the GC is not recommended. This might lead to memory being pushed into higher "generations" which are not often collected, thereby wasting more memory than gained in the long run.

David Schmitt
Nope, no hidden references to the dictionary.
Chris
Your comment about pushing objects into higher generations, care to elaborate? That sounds like an issue to be aware of.
Chris
Hang on, i thought that if you call gc.collect with no args, it simply collects _all_ generations? Wouldn't that rule out this issue with pushing stuff into higher generations? Thanks.
Chris
GC.Collect() does collect all gens. There are two issues here. 1) call it once: still referenced memory is pushed up, thus delaying normal collection, and 2) calling it regularly wastes more cpu because the gc has to process all gens everytime. So whatever you do with GC.Collect(), you loose. Except, of course, in those special cases where those effects don't matter to you.
David Schmitt
A: 

Edit:

To be fair, setting the reference to null does not free the memory, it assigns it's container to a different address, in this case null. According to MSDN, calling Clear(), does this: "The Count property is set to 0, and references to other objects from elements of the collection are also released. The capacity remains unchanged."

...

You should never call the garbage collector. You are using managed objects with no native resources, trust the garbage collector to clean up after you.

Besides the size of your dictionary, you don't need to worry about memory, memory isn't your problem, it's garbage collectors problem.

Calling Clear() will remove the references to any contained object inside, but the capacity remains unchanged.

On a technical note, collecting memory is expensive and a considerable time consuming operating. Reason being, not only does the GC handle heap memory and clean up your heap, it also sort of defrags the heap. It tries to move memory into contiguous blocks to speed up allocation for when some piece of code makes a large request.

p.s. How big is your dictionary that you use 155MB of memory?

Chris
Its big, i'm reconciling a million transactions.
Chris
holy crap that is a lot of transactions. I'm assuming this is a one time thing, or do you need to do this regularly?
Chris
Once a day, every day.
Chris
Actually i got that wrong - about half a dozen times each day.
Chris
OK, if that's the case, is there a reason it must be done in code? Why not just use the database to process these transactions, is there a reason not to?
Chris
+7  A: 

Private bytes reflect the process' memory usage. When objects are collected the associated memory segment may or may not be freed to the OS. The CLR manages memory at the OS level and since allocating and freeing memory isn't free there's no reason to free each piece of memory immediately as chances are that the application will probably request more memory later.

Brian Rasmussen
That all sounds reasonable, but it really _does_ free the memory to the OS if i call Txns.Clear(). Plus, i really want to be nice to the OS memory, since i'm not the only one using this server.
Chris
I guess what i'm asking is, why does txns.Clear() give memory back to the OS, when deferencing the entire dictionary doesn't give it back?
Chris
The point is that in a managed application you're one step removed from dealing directly with OS memory as the CLR does that for you. You may be able to construct scenarios where you can observe a direct effect as you describe, but that is not guaranteed to work as you depending on an implementation details of the CLR.
Brian Rasmussen
A: 

Windows has two memory availability events. I'd expect the CLR to respond to this. If there's plenty of memory available, the smart thing is to NOT run the garbage collector. So, to make sure that you are indeed observing bad CLR behavior, please repeat this test with another dummy application using a big heap of memory.

MSalters
+1  A: 

Do you need the memory back? The memory is available, it's just not being reclaimed. You should not clear the dictionary, hold a weak reference to it and let the runtime do its job.

If you want to see really deep into what's going on, check out the .NET Memory Profiler. That will give you visibility into exactly what's happening with your object, what generation it is, what memory is being used by what and on and on. :)

JP Alioto
I don't *need* the memory back, however in our situation i'd prefer to give it back to the OS quickly, if possible. I guess this question is becoming more of a theoretical 'how to do this?' rather than a 'i really need to do this' thing...
Chris