C# Free memory allocated by operator new from p/invoke DLL

Question

Hi, I have a C# application which is using a third party (closed source) native DLL through p/invoke.

During some of these calls, the DLL will allocate memory using operator new / operator new[] and return a pointer to use for reading the results.

This memory is never freed by the DLL after being allocated and returned. How can I perform the equivalent of a native operator delete / operator delete[] in C#?

Answer 1

I don't know how to do this explicitly, but you could wrap the DLL in Managed C++/CLI wrapper that has this facility for you.

Answer 2

If it does a global ::operator new you may be able to P/Invoke ::operator delete, although you'll have to use the mangled name for it (and the right CRT dll of course). If it does something like AClass *p = new AClass, I would not recommend trying to duplicate the effects of delete p with P/Invoke, it is going to be a little rough to get right (especially if they do anything interesting at all like overload new/delete). I would suggest using C++/CLI to wrap some of this stuff instead if it gets hairy (and frankly I think calling any delete via P/Invoke is probably hairy enough that it is worth breaking out the C++/CLI to save your sanity).

Answer 3

I do not know if you can do this with unmanaged code, but a separate app domain may be the answer. Load it up in its own memory space, and just dump the whole app domain when you're done.

Answer 4

Not possible. The DLL stores the handle returned by HeapCreate() internally. You'd have to know that handle to release the memory, you cannot get it out of the DLL. And you would have to know how many extra bytes were allocated by the DLL's malloc function to adjust the pointer.

You cannot even do this reliably if you write a wrapper for the DLL so that you can call free(). The DLL will have to use the DLL version of the CRT and you have to compile the wrapper with the exact same version of the compiler and the CRT so they'll share the same CRT DLL.

If you are really desperate, you could try to hack through GetProcessHeaps(). Although that's hard to get right, error prone and you really have to know what you're doing. DLLs returning pointers that need to be freed is a really bad practice even in C or C++. Just not in .NET, hail the garbage collector :)