Why does new / malloc fail on Win x64 although there is plenty of free RAM?

Question

I have a strongly recursive function, that creates a (very small) std::multimap locally for each function instance using new (which recurses to malloc/calloc in the std lib). After some hundred recursions new fails although i am using a native 64Bit application on Windows XP x64. The machine has 10 GB RAM, The application only uses about 1GB. No other big apps are running.

This happens a few minutes after starting the program and starting the recursive function. The recursive function has been called about 150.000 times at this point with a probably max. recursion of some hundreds. The problem occurring is not a stack overflow.

I am using Visual Studio 2005 and the dinkumware STL. The fault occurs in a release build.

EDIT: Ok, here is some code. I rearranged the code now and put the map on the stack, but it uses new to initialize - there it fails. I also tried with a std::multimap instead of hash_multimap. All of this die not change the behavior.

    int TraceBackSource(CalcParams *CalcData, CKnoObj *theKno, int qualNo, 
                            double maschFak, double partAmount, int MaschLevel, char *MaschID, 
                            double *totalStrFlow, int passNo, 
                            CTraceBackData *ResultData)

{   typedef std::hash_multimap<double, CStrObj *>StrFMap;
    StrFMap thePipes;

        for(...)
        {  
            ...
            thePipes.insert(std::make_pair(thisFlow, theStr));
        }

        // max. 5 elements in "thePipes"
        for(StrFMap::iterator it = thePipes.begin(); it != thePipes.end(); it++)
        {
            ...
            try
            {
                TraceBackSource(CalcData, otherKno, qualNo, maschFak * nodeFak, nodeAmount, SubMaschlevel, newMaschID, totalStrFlow, passNo, ResultData);
            }
            catch(std::exception &it)
            {   
                    Trace(0, "*** Exception, %s", it.what()); 
                    return 0;
            }

            return 0;
        }
}

Interestingly, the first failure runs into the catch handler, quite a bit later on i end with a ACCESS VIOLATION and a corrupted stack.

Answer 1

Your application is probably suffering from memory fragmentation. There might be plenty of memory available, but it may be fragmented into much smaller contiguous blocks than your application asks for.

Answer 2

The amount of RAM on your machine and the other processes running are irrelevant for this particular scenario. Every process has the same amount of virtual address space assigned to it. The size of this space is irrespective of the amount of RAM on your machine or other processes running.

What's happening here is likely one of the following

• You've simply allocated too much memory. Hard to do in 64 bit yes but possible
• There is no contiguous block of memory available which has the requested size.

Answer 3

Your number suggests an easily defaulted 1MB stacks size (c150K x 8 ). So from a quick look at your code (and that map::insert especially and not providing the for'...' code ) you are running into an interaction with stackoverflow.com :)

You are probably hitting it for the OS you're running it on. On Windows use the VS linker setttings or use editbin.exe or some exotic unportable api, triple your stack size and see whether it significantly changes the observed recursive count at time of exception.

Answer 4

As Majkara mentions, the thread stack space is a fixed size, and you are running out of it - it doesn't matter how much memory you have free. You need to rewrite your algorithm to be iterative using a stl::stack allocated on the heap (or some other data structure) to keep track of the depth.