Running a .NET application on Windows Server 2008 x64 with 16GB of RAM. This application needs to fetch and analyze a very large amount of data (about 64GB), and keep it all in memory at one time.
What I expect to see: Process size expands past 16GB to 64GB. Windows uses virtual memory to page the extra data to/from disk as needed. This is the classic virtual memory use case.
What I actually see: Process size is limited to the amount of physical memory (16GB). Application spends 99.8% of its time in the garbage collector.
Why is our application failing to use virtual memory? Is this a problem in the configuration of the .NET garbage collector, or in the Windows x64 virtual memory manager itself? What can I do to get our application to use virtual memory rather than be limited to physical memory?
Thanks.
-- Brian
Update: I have written a very small program that exhibits the same behavior:
using System;
namespace GCTest
{
class Program
{
static void Main()
{
byte[][] arrays = new byte[100000000][];
for (int i = 0; i < arrays.Length; ++i)
{
arrays[i] = new byte[320];
if (i % 100000 == 0)
{
Console.WriteLine("{0} arrays allocated", i);
System.Threading.Thread.Sleep(100);
}
}
}
}
}
If you want to try it, make sure to build for x64. You may have to modify the constants a bit to stress your system. The behavior I see is that the process bogs down as it approaches a size of 16GB. There is no error message or exception thrown. Performance monitor reports that the % of CPU time in GC approaches 100%.
Isn't this unacceptable? Where's the virtual memory system?