How can I detect message boxes popping up in another process?

Question

I'd like to execute some code whenever a (any!) message box (as spawned by the MessageBox Function) is shown in another process. I didn't start the process I'm monitoring.

I can think of three approaches:

1. Install a global CBT Hook procedure which tells me whenever a window is created on the desktop. Then, check whether the window belongs to the process I'm monitoring and whether the class name is #32770 (which is the class name of dialogs according to the About Window Classes page at the MSDN). This would probably work, but it would pull the DLL which contains the hook procedure into virtually every process on the desktop, and the hook procedure gets called a lot. It smells like a potential perfomance problem.
2. Try to subclass the #32770 system window class (is this possible at all?) and look for WM_CREATE messages in my custom window procedure.
3. Intercept the MessageBox Function API call (even though the remote process is running already!) and call my code from the hook function.

So far, I only know that the first idea is feasible, but it seems really inefficient. Can anybody think of a simpler solution than that to this problem?

Answer 1

I think: Approach 2 is impossible. Approaches 1-3 require dll, that is loaded into all processes, and approach 3 is "more right". I suppose searching windows by timer is not suite by some reasons?

Answer 2

I would go with the first option. You should be able to get away with only installing the hook for the main UI thread of the app you're monitoring, but if that doesn't work, even global CBT hooks aren't terribly resource intensive in my experience. Of course, you'll want your hook DLL to contain as little as possible other than the hook logic itself. If you need anything bulky, link it dynamically only when you know you're in the right process.

Answer 3

I can't think of any efficient solution that doesn't involve injecting code into the other process (this is basically what many types of hooks do by the way). But if you are willing to go down that path, you can intercept calls to MessageBox.

Spend some time stepping through into a call to MessageBox in the debugger in assembly language mode and you'll see that it's an indirect call through a lookup table (that's how exports work). so if you can get your code into the process, you can patch the table to jump to your code instead.

See http://www.codeproject.com/KB/threads/completeinject.aspx for code showing how to inject a dll into another process.