Out of memory (?) problem on Win32 (vs. Linux)

I have the following problem:

A program run on a windows machine (32bit, 3.1Gb memory, both VC++2008 and mingw compiled code) fails with a bad_alloc exception thrown (after allocating around 1.2Gb; the exception is thrown when trying to allocate a vector of 9 million doubles, i.e. around 75Mb) with plenty of RAM still available (at least according to task manager).

The same program run on linux machines (32bit, 4Gb memory; 32bit, 2Gb memory) runs fine with peak memory usage of around 1.6Gb. Interestingly the win32 code generated by mingw run on the 4Gb linux machine under wine also fails with a bad_alloc, albeit at a different (later) place then when run under windows...

What are the possible problems?

Heap fragmentation? (How would I know? How can this be solved?)
Heap corruption? (I have run the code with pageheap.exe enabled with no errors reported; implemented vector access with bounds checking --- again no errors; the code is essentially free of pointers, only std::vectors and std::lists are used. Running the program under Valgrind (memcheck) consumes too much memory and ends prematurely, but does not find any errors)
Out of memory??? (There should be enough memory)

Moreover, what could be the reason that the windows version fails while the linux version works (and even on machines with less memory)? (Also note that the /LARGEADDRESSAWARE linker flag is used with VC+2008 if that can have any effect)

Any ideas would be much appreciated, I am at my wits end with this... :-(

Tried with Release, Release with Debug info, Debug to no avail

Jonathan L. Verner 2009-10-24 13:14:22

I don't see why compiling in `Debug` mode an allocation will generate a huge amount of debugging data? If you are using the debugging allocator a few extra bytes will be used around the allocated block to track it and extra CPU cycles will be burned to write and check for specific bit patterns in free memory. Other that that it shouldn't affect how memory is allocated.

Martin Liversage 2009-10-24 14:00:59

@Martin, just something I have experienced several times with double arrays with Visual C, I didn't check into the arcanes of how Microsoft was managing it. The symptoms were exactly the same and it is a very quick test so it was worth mentioning, even if ultimately it didn't work for the OP in this case.

RedGlyph 2009-10-24 14:17:46

That's also a good point, virtual memory is what really matters in this case.

RedGlyph 2009-10-24 12:43:03

Thanks for the reply! How do I find out the virtual memory the process is using? Task Manager says the program is using 1.2Gb when it fails. This should be well below even the 2GB mark (and I am linking with LARGEADDRESSAWARE). At that point the program is trying to allocate only around 75Mb... Is there a way to know whether the address space is badly fragmented? How can it be avoided?

Jonathan L. Verner 2009-10-24 13:22:24

Try enabling a column in task manager that's called something like "virtual memory" on XPs and something like "private(?) bytes" on vistas. Also consider using process explorer from sysinternals (AKA Rusinovich) which is a lot superior to task manager. Something like "perfmon" (which I believe comes with windows), could also help you see what exactly is going on with your machine's memory.

Dmitry 2009-10-24 13:48:04

LARGEADDRESSAWARE has no effect unless you add /3GB to your boot.ini

karunski 2009-10-24 14:11:53

You can download this utility from sysinternals: http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx And then check what is the biggest chunk of virtual memory you are having. That should give you an idea how much fragmentation has occured and whether the memory allocation will succeed or not.

Naveen 2009-10-24 16:49:17

Thanks for the sysinternals tip... Very nice tool. It does show something weird going on with my program: there are consecutive blocks of allocated memory each approx double the size of its predecessor going from 2Mb to 131Mb. At least I have a thing to investigate

Jonathan L. Verner 2009-10-24 18:01:30

My guess is that, this allocation is done by the vector. Assuming that you are using the default memory allocator for controlling vector's memory allocation, it might try to allocate more than what are you asking for. Note that capacity() of the vector need not be equal to size(). This is done for performance reasons so as to avoid constant reallocations and provide amortized constant time push_back() operation. But these allocations may become too much for the bigger vectors, for example if you are asking for 75MB it might try to allocate more than that.

Naveen 2009-10-25 05:07:53

Other thing that can be tried is to use std::deque instead of std::vector as deque provides better memory management deallocates the unused blocks of memory whenever the size of the deque reduces. Vector will not deallocate the memory when it's size is reduced although you can use the swap trick so as to force the vector release its internal unused memory.

Naveen 2009-10-25 05:11:59

@Naveen: Yes the vector allocator (or list?) is probably the culprit.@karunski: Thanks, I'll try to convince the person who runs the production machine to add the 3GB flag.

Jonathan L. Verner 2009-10-27 13:21:05

Hmm, thanks for the explanation with the /3GB boot flag, I wasn't aware of that. The 3/1 split for Linux would seem to indicate why the program runs under Linux.

Jonathan L. Verner 2009-10-24 18:04:16

ansaurus

tags:

views:

answers:

Out of memory (?) problem on Win32 (vs. Linux)

related questions