Closing a MessageBox automatically

Question

I have a third party encryption library, which may create a MessageBox if key creation fails. The failure can be caused by bad random number generation or other rarities, and in most cases, trying again will result in success. My code will attempt key creation up to three times before deciding it failed.

Now, the issue is that the program may be used with automation. If a MessageBox is created during automation, it will block the process forever, because there's nobody to click the 'OK' button.

Does anyone know of a way to catch when this message box is created and automatically close it?

Anything is fair game, as long as it's not something that will make security suites angry. This means no hooking or code tunneling.

In summary, I need to catch when a MessageBox is created and close it. The MessageBox's creation is outside of my control. Modifying the code at runtime is not acceptable.

Also, I've noticed there are some other similar questions, but they don't have the same requirements.

EDIT: Additional note, I can find the message box via searching through all windows until I find one with a matching title and then send it a WM_CLOSE message, but I don't think this is a great solution. I also have no guarantee that the message box has been/will be displayed, or how long after my call it will be displayed. It could display instantly, it could display 1200 ms later, or it could not display at all.

Answer 1

I have once "remote controlled" an application by sending mouse click events to some controls. I guess you would have to do this in a separate thread that is watching for Events if a window is opened. Pretty ugly but working...

Answer 2

You can hope that it will be found by GetForegroundWindow, but this may catch other applications. The more brute force way is to iterate over all windows with EnumWindows looking for something that has a caption or text equal to this shown by the library.

Answer 3

Just before you begin the encryption process, install a WH_CBT hook, and in its callback watch for an nCode of HCBT_CREATEWND. If you get a matching class name ('#32770 (Dialog)' ?) and a matching title either return a nonzero value from the callback, or if that doesn't work post a WM_CLOSE (or a BM_CLICK to a relevant button if selecting an option is necessary). Uninstall the hook after the process for not messing with every possible dialog your application pops up.

Answer 4

That sounds like bad design on the part of that library. Generally any sort of utility library (like encryption) has no business invoking any kind of GUI (unless you explicitly ask it to).

Is there possibly some configuration or setting in this library that could disable its use of message boxes?

If not, I'd suggest that you might want to investigate using a different library. After all, if the designers of this library have already made this kind of poor design decision once, then there may be other unfortunate surprises lurking in there.

Answer 5

Create a new thread. If your function fails and a Message Box is opened, obtain a handle to the message box by looping through the windows (GetTopWindow, GetNextWindow) and comparing the window's process id to the one returned from GetCurrentProcessId().

Or, you can avoid all the hard work and just hook the MessageBox API with detours. It's not very hard, and if you don't want to pay for detours, you can do it manually.

1. Call VirtualProtect and set the memory protection at MessageBox at PAGE_EXECUTE_READWRITE
2. Create a naked function, and use it as a trampoline.
3. Create a function identical in parameters to MessageBox (this will be your hook)
4. Create a jump from MessageBox to your hook function.