How to make a .NET process run out of memory without exhausting all system memory.

Question

The problem is simple I have a process, that does ETL on some xml files. We started getting really large xml files and I started getting OutOfMemoryExceptions.

Fixing the process is relatively simple. However, I'd like to make a unit test for my NUnit suite to make sure the process will continue to be able to handle really large files. However, actually running out of memory on my development workstation slows down my machine, and is time consuming. Storing a humongous test file in version control is also a bad idea. If I could artificially limit a process, thread or appdomain to only use a fixed amount of ram, lets say 128 megs, I could make a smaller unit test that would not bring my workstation, to its knees.

Any suggestions? Is their some unmanaged API I can P/Invoke?

Answer 1

I don't quite understand what you're getting at here. Let's suppose that somehow you created an artificially small "test environment" that couldn't allocate as-big chunks of memory, so it threw OutOfMemoryExceptions on smaller files. What have you gained? The unit test was supposed to test whether you can handle bigger files on a real system, right?

The important thing is presumably that you can handle files "as big as they need to be" on whatever system they're going to be running on, and there's no real way to unit test that outside of trying out that file on that system.

(A smaller, less important thing might be whether you handle OutOfMemoryExceptions gracefully, but you don't need to actually run out of memory to test that; just make your method throw an exception once in a while and observe that it does the right thing.)

Answer 2

Can't you use a mocking framework for the memory allocation and have it throw OutOfMemoryException as one of the tests?

Having said that though, if you really have run out of memory there's not much your application can safely do, but if you can at least fail gracefully your users will be grateful.

An example: I had a case in a previous job where we were displaying 3D models of factories in real-time. The models grew so large that when we tried to load textures we'd get out of memory failures. We managed to keep the application alive and rendering by making sure that the code coped with null pointers even though the rest of the code thought that there should be texture information there.

Answer 3

Mocking is best. Actually raising an OOM is by defininition not a unit test. When dealing with memory, you are dealing with load testing. If you read the links at the bottom of this email, you'll find real OOMs are dastardly hard to reproduce and debug in the best of cases. A contrived OOM exception is not the true cause of the exception, and thus no more interesting than a mock for testing.

Stick with a unit test using a mock for validation. If you still get OOMs, throw more memory on your server and make your process recycle/ restart more often.

Here is some interesting reading on OutMemoryExceptions I collected the last time I fought with them. Summary: OOMs occur when the system can't allocate the amount that you requested - which doesn't mean you are out of memory.

• MSDN ManagedLeaks
• CLR Inside Out
• MSDN Debuging
• Bugslayer column on AD Plus

Answer 4

It is pretty easy to cause out of memory exceptions in a process.

Just create a loop that allocates memory in blocks that are small enough to not be on the large object heap (but not too many that you have that cause the exception) and then you can try to open a smaller file and that will cause the opening of the file to not be able to allocate enough contiguous memory and you will get your OOM exception when opening your file without needing a huge file. Something like this...

List<byte[]> items = new List<byte[]>();
for (int i = 0; i < 10000; i++)
{
     byte[] c = new byte[160000];
     items.Add(c);
}

byte[] next = new byte[1000000000];

If you run the above code as is, you will get a OOM exception on the last line. But, if you comment out the loop first it will execute with no errors. You will probably have to tweak the loop a little to get it to make your file open fail every time, but you can do it. Just run the loop before your call to open your file in your test and you will have used up a large chunk of memory and your open should fail.

Also, you might want to look into setting the /3GB switch if it is an option for you. It is not always the right answer and it has drawbacks associated with it, but it changes the virtual memory split form 2GB/2GB to 1GB/3GB allowing your process access to more virtual address space. This will give you a little more breathing room in the size of files that you can open. Again, you should read about the downsides of doing this before going after this as a solution and make sure that it is worth it if it helps your situation.

Here is how to enable it on the server