An impasse with hooking calls to HeapAlloc for a memory tracking application.

Question

I am writing a memory tracking application that hooks all the calls to HeapAlloc using IAT patching mechanism. The idea is to capture all the calls to HeapAlloc and get a callstack.

However I am currently facing a problem with getting the callstack using DBGHELP Apis. I found that the dbghelp dll itself is linking to MSVCRT dll and this dependancy results in a recursive call. When I try to get a callstack for any of the calls from the target application, dbghelp internally calls some method from MSVCRT that again calls HeapAlloc. And since I have already patched MSVCRT it results in an infinite loop.

Has anyone faced this problem and solved it ? Is there any way out of this impasse?

Answer 1

What about using some real memory tracking products like GlowCode?

Answer 2

This is a standard problem in function interception code. We had a similar issue with a logging library that used shared memory for storing log level information, while the shared memory library had to log information.

The way we fixed it could be applied to your situation, I believe.

In your intercept code, maintain a static flag that indicates whether or not you're in the middle of an intercept. When your intercept is called and the flag isn't set, set the flag then do what you currently do, including calling DbgHelp, then clear the flag.

If your intercept is called while the flag is set, only call the back-end HeapAlloc code without doing any of the other stuff (including calling DbgHelp which is what's causing your infinite recursion).

Something along the lines of (pseudo-code):

function MyHookCode:
    static flag inInterceptMode = false
    if inInterceptMode:
        call HeapAlloc
        return
    inInterceptMode = true
    call DbgHelp stuff
    call HeapAlloc
    inInterceptMode = false
    return

function main:
    hook HeapAlloc with MyHookCode
    : : :
    return