I'm currently working on a legacy app (win32, Visual C++ 2005) that allocates memory using LocalAlloc (in a supplied library I can't change). The app keeps very large state in fixed memory (created at the start with multiple calls to LocalAlloc( LPTR, size)). I notice that in release mode I run out of memory at about 1.8gb but in debug it happily goes on to over 3.8gb. I'm running XP64 with the /3gb switch. I need to increase the memory used in the app and I'm hitting the memory limit in release (debug works ok). Any ideas?
Sounds like your Release build is also compiled as x86. If not, than there must be something in your code which treats pointer as signed 32-bit integers and this code is only active in Release.
How does the running out of memory manifests itself?
Also, there is no reason to use /3gb flag for XP64 when running 64-bit applications: it doesn't change anything in this scenario
Why do you need to allocate such memory size ?
Do you have memory leaks ?
I have some real solutions for memory leaks, just let me know.
One suggestion: have a look at the base addresses of the DLLs that get loaded into the process space in release and debug mode, and see if there is much difference. It's possible that, in the release case, there are DLLs loaded at addresses so that, while there's enough free space in total to support a LocalAlloc() call, there isn't enough continuous address space to satisfy it. (For a contrived example, suppose that there was a DLL loaded at 0x40000000 (1Gb), another at 0x80000000 (2Gb), and another at 0xC0000000 (3Gb). Even if these DLLs were really small, the process couldn't allocate more than 1Gb at a time, as there's no continuous block of address space left free that's big enough).
You could also get a variation on this problem if the memory allocations happened in a different order in debug and release, too.
You probably have the Debug configuration linking with /LARGEADDRESSAWARE and the Release configuration linking with /LARGEADDRESSAWARE:NO (or missing altogether).
Check Linker->System->Enable Large Addresses in the project's configuration properties.