Problems with 'cross process subclassing'

Question

Two questions.

1) I understand this must be an expected outcome but maybe someone can tell me what I'm doing wrong; I'm trying to subclass all window classes in a global hook and it works except that I'm not able to close shop as I should and when the program initially registering the hook unregisters the hook and exits, subclassed applications start crashing.

Here's how I'm trying to do it..

// stores original wndprocs. In the hook dll, outside the shared memory.
map<HWND, WNDPROC> origWndProcs;

// in an EnumWindows callback, executed for all HWND's, also in the hook dll (UWM_REMOVE_HOOK is a registered unique message)
SendMessageTimeout(hWnd, UWM_REMOVE_HOOK, 0, 0, SMTO_ABORTIFHUNG | SMTO_NORMAL, 15000, res);

// Still in the same hook, in the subclassing wndproc..
if (msg == UWM_REMOVE_HOOK) {
   if (origWndProcs.find(hwnd) != origWndProcs.end()) {
      SetWindowLongPtr(hwnd, GWL_WNDPROC, (LONG_PTR)origWndProcs[hwnd]);
   }
}

// clears the hook..
__declspec(dllexport) BOOL ClearHooks(HWND hWnd) {

    BOOL unhooked = UnhookWindowsHookEx(hook) &&
       UnhookWindowsHookEx(kb_hook) &&
       UnhookWindowsHookEx(mouse_hook) && 
       UnhookWindowsHookEx(cbt_hook);

    if(unhooked)
       hWndServer = NULL;
    return unhooked;
}

In DllMain I don't do anything on DLL_PROCESS_DETACH. Instead, ClearHooks() is called from the program originally registering the hooks and there only after the hook has sent a message signalling that it has executed the EnumWindows operation (restores original wndprocs, see above).

I subclass windows in a WndProc hook; all visible windows that receive a message and whose current wndproc is not the one in the dll, are subclassed.

Basically all (as far as I can tell) applications crash on exit despite the fact that windows do seem to get the wndproc set back to what it was when it was replaced. Anyone have a clue what I might be doing wrong?

2) I need this to intercept WM_MINMAXINFO and modify window maxsize whenever a window is maximized. Unfortunately I can't do this in the dll but I have to talk with a program to get the size info. So, what's the best way to talk to that window; I need it to pass back some info so I can modify the structure that came with the original WM_MINMAXINFO message. Will a structure in WM_COPYDATA preserve it's data until the call to SendMessageTimeout returns?

Thanks

Answer 1

There are plenty of pain points here. You assume that no other code will subclass the window. And that such code will un-subclass it in the right order. There is no right order, your hooking is quite asynchronous from the program's execution.

But, the workaround is simple enough. You are already hooking with SetWindowsHookEx, might as well do one more. WH_CALLWNDPROC or WH_CALLWNDPROCRET, depending on what you want to do.