OutOfMemoryException thrown while memory is still available

Question

I need to build a big list of strings and keep it in memory, however while building it an OutOfMemoryException is thrown. According to Resource Monitor I still have 1GB of memory available. I found this KB article addressing the issue, but it seems like it should have been fixed in framework 1.1 SP1 (I am using 3.5 sp1).

Can anybody shed some light on what happens behind the scenes? Does the .net framework limit how much memory can be used by a single process (on a 32 bit system)? If so I can see why, but what doesn't make sense is that the application is only using 1.6GB and there is still ~1GB left to the system.

Edit - For those who asked here is some more in-depth information:

I have a List (Yeah, I could use something else, but I am just prototyping right now.), I generate a random string by doing a Guid.NewGuid().ToString(), and throw it in the list. What I am trying to do is generate a list with as many items as I can fit into it, and test different methods of looking up a specific one. My first guess was that some fragmentation is going on, but I dropped everything except the code below, and it still happens. I don't think this little snippet could create a lot of fragmentation, but I'm probably wrong.

        List<string> blah = new List<string>();

        for (int i = 0; i < 50000000; i++)
        {
            blah.Add(Guid.NewGuid().ToString());
        }

Answer 1

I don't really know the specifics of this bug, but I did run into this, or something very similar, years ago. Essential we found there was a hard limit on the amount of GDI handles which your application was bound to. Something like 9,999. When you hit this limit the application would crash with an out of memory exception no matter how much memory was free.

So you could be doing something similar, but since you mentioned you are working with strings, I imagine you are fragmenting the heap and exceeding its capacity. If your strings are big enough, they are probably on the large object heap. I think the implementation for the LOH was very poor in early version of the framework. If memory serves me correctly, objects didn't get deallocated properly, making it very easy to run out of space on only the LOH.

See these links for some more information

http://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-large-object-heap/ http://msdn.microsoft.com/en-us/magazine/cc534993.aspx http://blogs.msdn.com/maoni/archive/2006/04/18/large-object-heap.aspx http://social.msdn.microsoft.com/Forums/en-US/clr/thread/08e6bd5f-613e-41ae-9ab1-b05c7ff2710f

Answer 2

The problem is probably not that you don't have the memory "available", but more likely that you've fragmented the memory so much that when you try to add an item to the list, and it must be resized, no single block of available memory can hold it.

This will also lead to OutOfMemoryException.

How big is the list, in other words, how many strings do you have in it, exactly or roughly, when you get the exception?

And how are you populating the list? Do you know in advance the number of items you're going to add? And if so, do you specify the capacity when you construct the list?

If you don't know, would it be possible to figure that out, so you could specify that capacity?

Answer 3

Win32 has a 2GB per process limit on memory. Well, 4GB actually, but 2GB is reserved for kernel-mode, unless you have the /3GB OS startup option set.

Answer 4

The default constructor of List<T> starts with an empty array as internal storage. A call to Add(T item) checks if an array resize is necessary, and will double the Capacity property (via the EnsureCapacity method). When the Capacity property is set it ...

1. Creates a new array with the new size (double the original!)
2. Copies elements from the old one into the new one.
3. Changes internal reference into the new one.

So, assuming a Capacity of n would imply that during the resize you have a total memory usage of 3n.

How many items does your List contain when it fails? Try using the constructor that accepts an capacity if you know how many items will be added. This avoids any resizes that may need to be done (and may throw your OutOfMemoryException directly if you require huge memory blocks).