Can I switch the Visual C++ runtime to another heap?

Question

My program uses a third party dynamic link library that has huge memory leaks inside. Both my program and the library are Visual C++ native code. Both link to the Visual C++ runtime dynamically.

I'd like to force the library into another heap so that all allocations that are done through the Visual C++ runtime while the library code is running are done on that heap. I can call HeapCreate() and later HeapDestroy(). If I somehow ensure that all allocations are done in the new heap I don't care of the leaks anymore - they all go when I destroy the second heap.

Is it possible to force the Visual C++ runtime to make all allocations on a specified heap?

Answer 1

Sorry my last answer got posted half-baked, i pressed tab and enter without remembering this was a text box and not an editor...

Anyway heres it in full :

You can use the detours library to hook the allocation and deallocation functions, replace them with your own :

Vaguely something like this :

//declare a global 
HANDLE g_currentHeap;

LPVOID WINAPI HeapAlloc(HANDLE hHeap, DWORD dwFlags, SIZE_T dwBytes) 
{ 
    return OriginalHeapAlloc(g_currentHeap, dwFlags, dwBytes);
}


BOOL WINAPI HeapFree(HANDLE hHeap, DWORD dwFlags, LPVOID lpMem)
{
    return OriginalHeapFree(g_currentHeap, dwFlags, lpMem);
}

in the application load

HANDLE g_Heaps[2];

int main()
{
    // Two heaps
    g_Heaps[0] = HeapCreate(...);
    g_Heaps[1] = HeapCreate(...);


    // Do whatevers needed to hook HeapAlloc and HeapFree and any other heap functions 
    // and redirect them to the versions above
    // Save the old function pointers so we can call them
}

Then everytime you call an API from the 3rd party DLL you can do this

void someFn()
{
    g_currentHeap = g_Heaps[1];
    Some3rdPartyAPI();
    g_currentHeap = g_Heaps[0];

    SomeOtherFunction();

}

This should solve your problem

@peterchen : The C++ runtime calls HeapAlloc for new and malloc() so this approach will work. In fact I believe almost any languages runtime will use the win32 Heap functions, unless there was a special reason not to.

Answer 2

Redirecting only the DLL's allocations is tricky at best if both binaries link to it the same way.

The most robust way I can think of is moving the DLL into a separate process. That's fairly easy for a COM DLL that uses only IDispatch interfaces or provides a proxy/stub DLL. You would need to write a custom wrapper otherwise - depending on the DLL's API that's a lot of work or might be a performance problem.

If require to remain in-process, you could hook CRT allocations, and redirect allocations made by the library to another allocator (such as a Win32 heap).

The mine/theirs decision would safest be made by wrapping all calls to the library that sets a global flag. Alternatively, you could inspect thecall stack - but that won't work in all scenarios. With both solutions, watch out for callback implemented in your code but called by the library.

[edit] _CRTSetAllocHook does work in debug buils only, though.