P-invoke call fails if too much memory is assigned beforehand

Question

I've got a p-invoke call to an unmanaged DLL that was failing in my WPF app but not in a simple, starter WPF app. I tried to figure out what the problem was but eventually came to the conclusion that if I assign too much memory before making the call, the call fails. I had two separate blocks of code, both of which would succeed on their own, but that would cause failure if both were run. (They had nothing to do with what the p-invoke call is trying to do).

What kind of issues in the unmanaged library would cause such an issue? I thought that the managed and unmanaged heaps were supposed to be automatically separated.

The crash as far as I can tell is happening in a dynamically loaded secondary DLL from the one p-invoked into. Could that have something to do with it?

Answer 1

A shot in the dark given there is not much info to work with.

Is it possible that the unmanaged DLL needs to be loaded at a specific base address and when you allocate too much memory or other assemblies are loaded, the DLL is not able to load at the correct address.

http://msdn.microsoft.com/en-us/library/w368ysh2.aspx

Answer 2

Unmanaged code is prone to corrupt the heap. The side effects of that corruption are very unpredictable, it depends on what happens afterwards with that corrupted memory. It is not uncommon that nothing bad happens if the corruption is not in a crucial location. Changing the memory allocation pattern of your program can change that outcome.

All you really know right now is that the unmanaged code can't be trusted. Doing something about it is invariably hard, especially from a managed host program. You won't get anywhere until you start writing unit tests for that unmanaged code, using unmanaged code to exercise it, and find a reproducible bomb that you could tackle with an unmanaged debugger.