views:

121

answers:

2

Running a .NET application on Windows Server 2008 x64 with 16GB of RAM. This application needs to fetch and analyze a very large amount of data (about 64GB), and keep it all in memory at one time.

What I expect to see: Process size expands past 16GB to 64GB. Windows uses virtual memory to page the extra data to/from disk as needed. This is the classic virtual memory use case.

What I actually see: Process size is limited to the amount of physical memory (16GB). Application spends 99.8% of its time in the garbage collector.

Why is our application failing to use virtual memory? Is this a problem in the configuration of the .NET garbage collector, or in the Windows x64 virtual memory manager itself? What can I do to get our application to use virtual memory rather than be limited to physical memory?

Thanks.

-- Brian

Update: I have written a very small program that exhibits the same behavior:

using System;

namespace GCTest
{
    class Program
    {
        static void Main()
        {
            byte[][] arrays = new byte[100000000][];
            for (int i = 0; i < arrays.Length; ++i)
            {
                arrays[i] = new byte[320];
                if (i % 100000 == 0)
                {
                    Console.WriteLine("{0} arrays allocated", i);
                    System.Threading.Thread.Sleep(100);
                }
            }
        }
    }
}

If you want to try it, make sure to build for x64. You may have to modify the constants a bit to stress your system. The behavior I see is that the process bogs down as it approaches a size of 16GB. There is no error message or exception thrown. Performance monitor reports that the % of CPU time in GC approaches 100%.

Isn't this unacceptable? Where's the virtual memory system?

+2  A: 

It sounds like you're not keeping a reference to the large data. The garbage collector will not collect referenced objects.

Stephen Cleary
I don't think that's relevant. None of the data is actually garbage - it's all referenced. Thus, I'm not expecting the GC to collect any memory at all.It appears that the GC is consuming 100% of CPU because the process has run out of physical memory and .NET is trying to free some. However, there's nothing to be freed - ideally, the process should just expand using virtual memory instead of spending all its time in the GC trying to free (non-existent) unused objects.
brianberns
If you're allocating 64 GB worth of data, but only 16 GB is actually allocated, then the rest of it isn't being referenced and has been GC'ed. Unless I'm not understanding your question.
Stephen Cleary
You're not understanding the question. I'm trying to allocate 64GB of data, but the process bogs down after I've only allocated 16GB of data. I never get a chance to allocate the rest of the data.
brianberns
Even if it's not garbage, the collector still has to walk through 16GB of data to decide what's in and what's ready to throw out which would cause it to potentially touch paged data and end of in the "death spiral" Stephen describes. Have you tried using the server GC version (mscorsvr.dll) instead of the default workstation version (mscorwks.dll)?
Paolo
@Paolo I stupidly updated the wrong post. See my update. I did try a lot of things including: forcing server GC, low latency GC, changing working set sizes, forced GC - they did not make any appreciable difference.
chibacity
+4  A: 

Have you checked to make sure that your paging file is configured so that it can expand to that size?

Update

I've been playing around with this quite a bit with your given example, and here's what I see.

System: Windows 7 64bit, 6GB of triple-channel RAM, 8 cores.

  1. You need an additional paging file on another spindle from your OS or this sort of investigation will hose your machine. If everything is fighting over the same paging file, it makes things worse.

  2. I am seeing a large amount of data being promoted from generation to generation in the GC, plus a large number of GC sweeps\collections, and a massive amount of page faults as a result as physical memory limits are reached. I can only assume that when physical memory is exhausted\very high, that this triggers generation sweeps and promotions thus causing a large amount of paged-out memory to be touched which is leading to a death spriral as touched memory is paged in and other memory is forced out. The whole thing ends in a soggy mess. This seems to be inevitable when allocating a large number of long-lived objects which end up in the Small Object Heap.

Now compare this to allocating objects in a fashion will allocate them directly into the Large Object Heap (which does not suffer the same sweeping and promotion issues):

private static void Main()
{
    const int MaxNodeCount = 100000000;
    const int LargeObjectSize = (85 * 1000);

    LinkedList<byte[]> list = new LinkedList<byte[]>();

    for (long i = 0; i < MaxNodeCount; ++i)
    {
        list.AddLast(new byte[LargeObjectSize]);

        if (i % 100000 == 0)
        {
            Console.WriteLine("{0:N0} 'approx' extra bytes allocated.",
               ((i + 1) * LargeObjectSize));
        }
    }
}

This works as expected i.e. virtual memory is used and then eventually exhausted - 54GB in my environment\configuration.

So it appears that allocating a mass of long-lived small objects will eventually lead to a vicious cycle in the GC as generation sweeps and promotions are made when physical memory has been exhausted - it's a page-file death spiral.

Update 2

Whilst investigating the issue I played with a number of options\configurations which made no appreciable difference:

  • Forcing Server GC mode.
  • Configuring low latency GC.
  • Various combinations of forcing GC to try to amortize GC.
  • Min\Max process working sets.
chibacity
No, but shouldn't the Windows virtual memory manager automatically start paging, at least to some degree past 16GB? What's the point of having virtual memory if it's limited by the amount of physical memory?
brianberns
It is configurable - worth checking. Problems could range from an unexpected policy, to some bright spark turning paging off.
chibacity
+1, very important. A process can never allocate more virtual memory then will fit in the paging file.
Hans Passant
OK, I increased the pagefile to 30GB, but it had no effect. I think the .NET garbage collector is preventing me from allocating more than 16GB worth of objects, so the Windows virtual memory system never has a chance to start paging.
brianberns
Thank you very much for your updates. The problem for me, of course, is that my objects are naturally smaller than LargeObjectSize (as are most objects in a typical OO application).Are you seeing actual paging to/from the OS paging file? My hunch is that it's not happening that way. Instead, I suspect that the .NET garbage collector "panics" as physical memory is exhausted, and thus is constantly interrupting the program to look for unreferenced objects that it can deallocate. As a result, the process never even gets a chance to allocate any memory beyond the amount of physical RAM present.
brianberns
@brianberns Yes I am definitely seeing a massive amount of hard page faults\page file activity. You can see this in perfmon. I am assuming that a [Low Memory Situation](http://msdn.microsoft.com/en-us/magazine/cc534993.aspx) is being triggered by the OS is causing GC activity and that then this is causing the page-file death spiral as generations are scanned and so memory is paged in and forced out right on the very limits.
chibacity
@brianberns I hear you on the typical object size issue, but I'm confident in my answer, so it would seem unavoidable. If you think about what you are trying do, then adding 64GB of data to the Small Object Heap does seem at odds - you don't want your data to be garbage collected. At least with the LOH example you have a new area to explore.
chibacity