Problem with window subclassing: endless cycle

Question

I have an application written on MS VC++ 6. It performs subclassing for several windows of controls (I needed this for custom skinning mechanism). New window procedure calls window procedure (its address was saved during the subclassing operation) in the end of its body - using CallWindowProc, of course.

It worked Ok for years until one of my customers (WinXP) complained that the application fails right after start. What I got crash dump I saw: stack overflow happened. Like this:

user32.dll!__SEH_prolog() + 0x1b bytes

user32.dll!_CallWindowProcAorW@24() + 0x51 bytes

user32.dll!_CallWindowProcW@20() + 0x1b bytes

MyModule.dll!CSkinManager::SkinWindowProc(HWND__ * hWnd=0x00150526, unsigned int mess=70, unsigned int wParam=0, long lParam=1230632) Line 7120 C++

user32.dll!_InternalCallWinProc@20() + 0x28 bytes

user32.dll!_UserCallWinProcCheckWow@32() + 0xb7 bytes

user32.dll!_CallWindowProcAorW@24() + 0x51 bytes

user32.dll!_CallWindowProcW@20() + 0x1b bytes

02ef007c()

user32.dll!_InternalCallWinProc@20() + 0x28 bytes

user32.dll!_UserCallWinProcCheckWow@32() + 0xb7 bytes

user32.dll!_CallWindowProcAorW@24() + 0x51 bytes

user32.dll!_CallWindowProcW@20() + 0x1b bytes

MyModule.dll!CSkinManager::SkinWindowProc(HWND__ * hWnd=0x00150526, unsigned int mess=70, unsigned int wParam=0, long lParam=1230632) Line 7120 C++

user32.dll!_InternalCallWinProc@20() + 0x28 bytes

user32.dll!_UserCallWinProcCheckWow@32() + 0xb7 bytes

user32.dll!_CallWindowProcAorW@24() + 0x51 bytes

user32.dll!_CallWindowProcW@20() + 0x1b bytes

02ef007c()

....

user32.dll!_InternalCallWinProc@20() + 0x28 bytes

user32.dll!_UserCallWinProcCheckWow@32() + 0xb7 bytes

user32.dll!_CallWindowProcAorW@24() + 0x51 bytes

user32.dll!_CallWindowProcW@20() + 0x1b bytes

MyModule.dll!CSkinManager::SkinWindowProc(HWND__ * hWnd=0x00150526, unsigned int mess=70, unsigned int wParam=0, long lParam=1230632) Line 7120 C++

user32.dll!_InternalCallWinProc@20() + 0x28 bytes

user32.dll!_UserCallWinProcCheckWow@32() + 0xb7 bytes

user32.dll!_CallWindowProcAorW@24() + 0x51 bytes

user32.dll!_CallWindowProcW@20() + 0x1b bytes

02ef007c()

user32.dll!_InternalCallWinProc@20() + 0x28 bytes

Where MyModule.dll — name of my DLL in my application; CSkinManager::SkinWindowProc — name of that window procedure that I used in the subclassing. What is '02ef007c()' — I do not know. WinDbg displays '+0x02ef007b' instead of this - this says me nothing too. From the crash dump I saw: several DLLs were embedded into my process, for instance AME_SMTPSensor.dll of some software named 'Digital Guard' (something like anti-virus). Or penjpn.dll of 'Microsoft JPN Handwriting Input UI' (my customer is Japanese). Maybe '02ef007c()' point to code of such embedded DLL?

One way or another, endless end of window procedures happened. My windows procedure CSkinManager::SkinWindowProc in teh end of its work calls CallWindowProc(OldProcAdds), where OldProcAdds is previous window procedure. This OldProcAdds, situated in that mysterious '02ef007c()', in the end calls CSkinManager::SkinWindowProc. In some time the stack overflows.

My version is: some of the embedded DLLs subclassed window procedure of my window as soon as it was created. Then I subclassed it; and stored address of the previous window procedure. Right after this the embedded DLL detected this fact and AGAIN did the subclassing — I think it always want to be 'on the top'.During this its window procedure, like mine, saves address of previous window procedure (my CSkinManager::SkinWindowProc); and calls it in the end. We have endless cycle.

What can I do? Unfortunately I do not have access to that PC; I can only investigate dumps, logs etc.

Answer 1

As soon as the other dll has re-subclassed your window you are in for a world of pain. There is nothing you can do and you are entirely at the mercy of their buggy code :- because they re-subclassed the window, their old-proc is now you, and your old-proc is them.

The only way you can sanely get out of this situation would be to set a flag to detect recursion:

// declare a variable with static storage.
// Should make it thread local if theres any chance you are
// subclassing multiple thread's windows
__declspec(thread) int gRecursive=0;


// in your windowproc, inhibit the recursive call.
if(!gRecursive++){
   ret = CallWndProc(OldProc,...);
   gRecursive--; 
}
return ret;