c# 'System.OutOfMemoryException' was thrown when there is still plenty of memory free

Question

This is my code:

int size = 100000000;
double sizeInMegabytes = (size * 8.0) / 1024.0 / 1024.0; //762 mb
double[] randomNumbers = new double[size];

Exception: Exception of type 'System.OutOfMemoryException' was thrown.

I have 4GB memory on this machine 2.5GB is free when I start this running, there is clearly enough space on the PC to handle the 762mb of 100000000 random numbers. I need to store as many random numbers as possible given available memory. When I go to production there will be 12GB on the box and I want to make use of it.

Does the CLR constrain me to a default max memory to start with? and how do I request more?

Thanks

Update

I thought breaking this into smaller chunks and incrementally adding to my memory requirements would help if the issue is due to memory fragmentation, but it doesn't I can't get past a total ArrayList size of 256mb regardless of what I do tweaking blockSize.

private static IRandomGenerator rnd = new MersenneTwister();
private static IDistribution dist = new DiscreteNormalDistribution(1048576);
private static List<double> ndRandomNumbers = new List<double>();

private static void AddNDRandomNumbers(int numberOfRandomNumbers) {
    for (int i = 0; i < numberOfRandomNumbers; i++) {
      ndRandomNumbers.Add(dist.ICDF(rnd.nextUniform()));                
  }
}

From my main method:

int blockSize = 1000000;

while (true) {
  try
  {
    AddNDRandomNumbers(blockSize);                    
  }
  catch (System.OutOfMemoryException ex)
  {
    break;
  }
}            
double arrayTotalSizeInMegabytes = (ndRandomNumbers.Count * 8.0) / 1024.0 / 1024.0;

Answer 1

Increase the Windows process limit to 3gb. (via boot.ini or Vista boot manager)

Answer 2

You may want to read this: "“Out Of Memory” Does Not Refer to Physical Memory" by Eric Lippert.

Answer 3

You don't have a continuous block of memory in order to allocate 762MB, your memory is fragmented and the allocator cannot find a big enough hole to allocate the needed memory.

1. You can try to work with /3GB (as others had suggested)
2. Or switch to 64 bit OS.
3. Or modify the algorithm so it will not need a big chunk of memory. maybe allocate a few smaller (relatively) chunks of memory.

Answer 4

32bit windows has a 2GB process memory limit. The /3GB boot option others have mentioned will make this 3GB with just 1gb remaining for OS kernel use. Realistically if you want to use more than 2GB without hassle then a 64bit OS is required. This also overcomes the problem whereby although you may have 4GB of physical RAM, the address space requried for the video card can make a sizeable chuck of that memory unusable - usually around 500MB.

Answer 5

Well, I got a similar problem with large data set and trying to force the application to use so much data is not really the right option. The best tip I can give you is to process your data in small chunk if it is possible. Because dealing with so much data, the problem will come back sooner or later. Plus, you cannot know the configuration of each machine that will run your application so there's always a risk that the exception will happens on another pc.

Answer 6

I'd advise against the /3GB windows boot option. Apart from everything else (it's overkill to do this for one badly behaved application, and it probably won't solve your problem anyway), it can cause a lot of instability.

Many Windows drivers are not tested with this option, so quite a few of them assume that user-mode pointers always point to the lower 2GB of the address space. Which means they may break horribly with /3GB.

However, Windows does normally limit a 32-bit process to a 2GB address space. But that doesn't mean you should expect to be able to allocate 2GB!

The address space is already littered with all sorts of allocated data. There's the stack, and all the assemblies that are loaded, static variables and so on. There's no guarantee that there will be 800MB of contiguous unallocated memory anywhere.

Allocating 2 400MB chunks would probably fare better. Or 4 200MB chunks. Smaller allocations are much easier to find room for in a fragmented memory space.

Anyway, if you're going to deploy this to a 12GB machine anyway, you'll want to run this as a 64-bit application, which should solve all the problems.

Answer 7

If you need such large structures, perhaps you could utilize Memory Mapped Files. This article could prove helpful: http://www.codeproject.com/KB/recipes/MemoryMappedGenericArray.aspx

LP, Dejan

Answer 8

Rather than allocating a massive array, could you try utilizing an iterator? These are delay-executed, meaning values are generated only as they're requested in an foreach statement; you shouldn't run out of memory this way:

private static IEnumerable<double> MakeRandomNumbers(int numberOfRandomNumbers) 
{
    for (int i = 0; i < numberOfRandomNumbers; i++)
    {
        yield return randomGenerator.GetAnotherRandomNumber();
    }
}


...

// Hooray, we won't run out of memory!
foreach(var number in MakeRandomNumbers(int.MaxValue))
{
    Console.WriteLine(number);
}

The above will generate as many random numbers as you wish, but only generate them as they're asked for via a foreach statement. You won't run out of memory that way.

Alternately, If you must have them all in one place, store them in a file rather than in memory.