API Hook on a COM object function?

Question

Greetings StackOverflowians,

As discovered here, Windows 7 features a bug in which the DISPID_BEFORENAVIGATE2 event does not fire for Windows Explorer instances. This event allows shell extensions to be notified when a navigation is about to take place, and (most importantly for me) have the opportunity to cancel the navigation. I've been looking for a workaround for quite some time, and I think I found one. But, I'd like to get some opinions on how safe it is.

I've been playing with API hooking a lot lately, and I'm already using it to hook a few functions for my extension. I noticed that there is a function in IShellBrowser that controls navigation. At first I thought you couldn't hook something like that, but upon reading about the layout of a COM object I realized it should be possible by just grabbing the right function pointer out of the vtable of any active instance. Sure enough, it works like a dream. After the hook is set, all navigations in all Explorer windows run right through my detour function, and I can decide whether to reject them based on their target pidl.

So my question is, is there any reason I should NOT do this? I've never heard of API hooking used to hook COM object functions. Are there circumstances it which it wouldn't work? Is it dangerous? (Any more than regular API hooking, at least)

The relevant code follows. I'm using MinHook, a minimalistic hooking library that uses the tried-and-true method of trampoline functions.

typedef HRESULT (WINAPI *BROWSEOBJECT)(IShellBrowser*, PCUIDLIST_RELATIVE, UINT);
HRESULT WINAPI DetourBrowseObject(IShellBrowser* _this, PCUIDLIST_RELATIVE pidl, UINT wFlags);
BROWSEOBJECT fpBrowseObject = NULL;
BROWSEOBJECT ShellBrowser_BrowseObject = NULL;

bool Initialize() {
    if(MH_Initialize() != MH_OK) {
        return false;
    }

    // Get a reference to an existing IShellBrowser.  Any instance will do.
    // ShellBrowser enum code taken from The Old New Thing
    IShellWindows *psw;
    BOOL fFound = FALSE;
    if (SUCCEEDED(CoCreateInstance(CLSID_ShellWindows, NULL, CLSCTX_ALL, IID_IShellWindows, (void**)&psw))) {
        VARIANT v;
        V_VT(&v) = VT_I4;
        IDispatch  *pdisp;
        for (V_I4(&v) = 0; !fFound && psw->Item(v, &pdisp) == S_OK; V_I4(&v)++) {
            IWebBrowserApp *pwba;
            if (SUCCEEDED(pdisp->QueryInterface(IID_IWebBrowserApp, (void**)&pwba))) {
                IServiceProvider *psp;
                if (SUCCEEDED(pwba->QueryInterface(IID_IServiceProvider, (void**)&psp))) {
                    IShellBrowser *psb;
                    if (SUCCEEDED(psp->QueryService(SID_STopLevelBrowser,IID_IShellBrowser, (void**)&psb))) {
                        fFound = true;

                        // Grab the 11th entry in the VTable, which is BrowseObject
                        void** vtable = (*(void***)(psb));
                        ShellBrowser_BrowseObject = (BROWSEOBJECT)(vtable[11]);
                        psb->Release();
                    }
                    psp->Release();
                }
                pwba->Release();
            }
            pdisp->Release();
        }
        psw->Release();
    }

    if(fFound) {
        if(MH_CreateHook(ShellBrowser_BrowseObject, &DetourBrowseObject, reinterpret_cast<void**>(&fpBrowseObject)) != MH_OK) {
            return false;
        }
        if(MH_EnableHook(ShellBrowser_BrowseObject) != MH_OK) {
            return false;
        }
    }
    return true;
}

HRESULT WINAPI DetourBrowseObject(IShellBrowser* _this, PCUIDLIST_RELATIVE pidl, UINT wFlags) {
    if(NavigateIsOkay(pidl, wFlags)) {
        return fpBrowseObject(_this, pidl, wFlags);
    }
    else {
        return S_FALSE;
    }    
}

Answer 1

I've never heard of API hooking used to hook COM object functions.

Member functions of COM Objects are not really that different and can actually be hooked just fine if you stick to the usual guidelines for hooking. A few years ago, I had to hook COM components of a proprietary CRM solution to connect it to a database server. The application worked fine and has been running quite stable for several years.