.NET out of memory troubleshooting

Question

After reading a few enlightening articles about memory in the .NET technology, Out of Memory does not refer to physical memory, 597499.

I thought I understood why a C# app would throw an out of memory exception -- until I started experimenting with two servers-- both are having 2.5 gigs of ram, windows server 2003 and identical programs running.

The only significant difference between the two being one has 7% hard drive storage left and the other more than 50%.

The server with 7% storage space left is consistently throwing an out of memory while the other is performing consistently well.

My app is a C# web application that process' hundreds of MBs of String object.

Why would this difference happen seeing that the most likely reason for the out of memory issue is out of contiguous virtual address space.

thanks SO

Answer 1

All I can think of is that you're exhausting the virtual memory. Sounds like you need to run a memory profiler on the app.

I've used the Red Gate profiler in similar situations in the past. You may be surprised how much memory your strings are actually using.

Answer 2

Is the paging file fragmentation different on each machine? High fragmentation could slow down paging operations and thus exacerbate memory issues. If the paging file is massively fragmented, sort it out e.g. bring the server off-line, set the paging file size to zero, defrag the drive, re-create the paging file.

It's hard to give any specific advice on how to deal with perf problems with your string handling without more detail of what you are doing.

Answer 3

I would agree that your best bet is to use a memory profiler. I've used .Net Memory Profiler 3.5 and was able to diagnose the issue, which in my case were undisposed Regex statements. They have demo tutorials which will walk you through the process if you're not familiar.

As you your question, any single reference to the strings, the jagged array for instance, would still prevent the string from disposing. Without knowing more about your architecture, it would be tough to make a specific recommendation. I would suggest trying to optimize your app before extending memory though. It will come back to bite you later.

Answer 4

Why would this difference happen seeing that the most likely reason for the out of memory issue is out of contiguous virtual address space?

With 7% free hard disk your server is probably running out of space to page out memory from either your process or other processes, hence it has to keep everything in RAM and therefore you are unable to allocate additional memory more often than on the server with 50% free space.

What solutions do you guys propose?

Since you've already run a profiler and seen at least 600MB+ of usage with all the string data you need to start tackling this problem.

The obvious answer would be to not hold all that data in memory. If you are processing a large data set then load a bit, process it and then throw that bit away and load the next bit instead of loading it all up front.

If it's data you need to serve, look at a caching strategy like LRU (least recently used) and keep only the hottest data in memory but leave the rest on disk.

You could even offload the strings into a database (in-memory or disk-based) and let that handle the cache management for you.

A slighty left-of-field solution I've had to use in the past was simply compressing the string data in memory as it arrived and decompressing it again when needed using the SharpZipLib. It wasn't that slow surprisingly.

Answer 5

An OutOfMemoryException is more likely to indicate fragmentation in your page file - not that you are out of RAM or disk space.

It is generally (wrongly) assumed that the page file is used as a swap disk - that RAM overflow is written to the page file. All allocated memory is stored in the page file and only data that is under heavy usage is copied to RAM.

There's no simple code fix to this problem other than trying to reduce the memory footprint of your application. But if you really get desperate you can always try PageDefrag, which is a free application originally developed by SysInternals.