How can I make a window in another process modal?

Question

I have an application that calls some other utility application to set some settings for a particular device. That utility application is called using ShellExecuteEx.

So as not to confuse the user, it would be better to made the window of the utility application modal to my main window. How does one do this?

Things I've tried:

1. WaitForSingleObjectEx on the process after ShellExecuteEx, INFINITE TIMEOUT - window is modal, but main application does not repaint (because it's waiting for the single object!)
2. WaitForSingleObjectEx on the process after ShellExecuteEx, some small timeout, then call Peekmessage and DispatchMessage - repaint now works, but utility application is no longer "modal". The main application responds to mouse clicks, button clicks, etc
3. EnableWindow(FALSE), then do method #2, then EnableWindow(TRUE) - WORKS!!!, but after this, the z-order of my application changed. (it's now below some other window). why?!

Answer 1

The short answer is that there is no way to seamlessly make a window in thread B modal for a window in thread A, even if the threads are in the same process. If you own the code for both windows, you may be able to come close, but in that case you will achieve much better results for the effort by putting all of your UI in one thread.

If you try to suggest to the user that thread B's window is modal for thread A's, there are a lot of subtle Z-order and activation behaviors you have to get right (as you have noticed) lest you suffer an uncanny-valley effect of sorts, where it's clear to the user that thread B's window is trying to be something it's not and therefore seems broken.

To avoid that, I would take this approach:

1. The user clicks on "FDA Inspection" in canner.exe's main window. canner.exe shows a modal dialog indicating that it is opening an external program ("Opening Botulism Settings..."). This disables the main window, etc. so that the user knows a modal interaction is taking place.
2. canner.exe calls ShellExecuteEx() to start botulism.exe.
3. canner.exe calls WaitForInputIdle() on the handle returned from ShellExecuteEx(). WaitForInputIdle() will return (approximately, but usually close enough) when botulsim.exe is ready for user interaction. If botulism.exe typically takes five or more seconds to show its UI, I may use a short timeout with WaitforInputIdle() in a loop and occasionally process any pending messages with PeekMessage()/ProcessMessage().
4. canner.exe changes its dialog text to reflect that it is waiting for the user to close botulism.exe ("Close Botulism Settings to continue...").
5. canner.exe calls MsgWaitForMultipleObjects() in a loop to wait until botulsim.exe closes. MsgWaitForMultipleObjects() will return when the handles passed are signaled or when there are messages waiting in the thread's queue.
6. If the user clicks the close box in canner.exe's modal dialog while canner.exe is waiting, canner.exe prompts the user that botulism.exe is still running ("Botulism Settings is still open, continue anyway?", "Yes, I know" or "No, I'm not done"). If confirmed, canner.exe closes the dialog and cancels the original FDA inspection started in step 1 and returns to the main window's message loop.
7. When MsgWaitForMultipleObjects() indicates that botulism.exe is finished, canner.exe closes the dialog and continues normally with the FDA inspection started in step 1.

This way, if everything proceeds normally and quickly, the interaction may well be seamless, but if something goes wrong with the child process or the Z-order gets changed, etc. it will be clear why the parent process is waiting and what the user needs to do to either cancel or go on with the task he started.

Answer 2

Let's take a look at your approach #3, which is very close to what you want. I suspect the problem is that when the secondary app closes, Windows decides that it doesn't want to restore focus to a disabled window. You could try to re-enable your window before that happens but that's likely to be tricky (and not worth the effort).

Instead of disabling the window directly, try disabling it by just ignoring user input. So rather than calling EnableWindow, change your message loop to filter out input messages. In particular, if

msg >= WM_KEYFIRST || msg <= WM_KEYLAST || msg >= WM_MOUSEFIRST || msg <= WM_MOUSELAST

then discard the message; otherwise, pass it on to the normal dispatch loop. What you're doing is creating your own disabled window, but Windows doesn't know that.

Answer 3

You have two things to simulate: ownership and modality.

To simulate ownership: You need to set the owner of your new child process window to your window. This should alleviate any z ordering issues. Though I don't know if this works from another process. If not then you might have to attach your thread input queues and then call it. Or use some other code injection technique.

SetWindowLong <target window handle>, GWL_HWNDPARENT, <new owner handle>

To simulate modality, I think you are on the right track with EnableWindow and the WaitForSingleObjectEx.

Answer 4

EnableWindow is correct, this is generally how message boxes and other "modal" windows do it. As for zorder changing, you can intercept the WM_WINDOWPOSCHANGING message and set the SWP_NOZORDER flag to prevent the zorder change. Make sure you only do this while you are setting EnableWindow(false).

Answer 5

Just logical suggestion,
Maybe you can create invisible modal form, and from him use the method #1.