How can I prevent a third party library from displaying a MessageBox?

Question

I am integrating a third party C-based SDK into my .NET application. The application will run as a Windows service on a server, so it should not interact with the user in any way.

Unfortunately, in certain error conditions it insists on calling MessageBoxA, presumably to report that something bad has happened. When this happens, the service stops responding. I am guessing that it is waiting for someone to press Ok?

It is not possible to have the vendor change their code for me.

Is there a way I can make this call into a no-op so my code can deal with the situation automatically?

EDIT: It may be important to mention that in my particular case the service would automatically restart if it crashed. A graceful (as possible) and sudden exit is probably the best resolution for a situation where a MessageBox is displayed in my case.

Answer 1

You could use an editor to find the location of this and remove the call from their binary. But that may or may not be allowable under the usage limitations with the software. Certainly if you re-distribute it it may cause problems - you should ask the vendor and report it as a defect and suggest that you have a workaround for your own use.

For people used to do this kind of thing (cracking licenses or other reverse engineering) this is pretty straightforward, but the real question is, what happens if it is ignored - does it still continue to work?

Answer 2

This is not very easy. I would think the only way is to write a message hook to trap messages pumped through all the windows message loops on the system. But I'm not even sure that a service can create a message hook given it does not have access to the default window station under its default credentials (LocalSystem). So task # one would be to see if that would actually work.

But here is a rough sketch of how I would try to do it outside of a service context and you can see if it applies:

1. Create a windows hook to trap messages from all windows message queues on the system.
2. Trap all WM_CREATE messages, and probe the CREATESTRUCT paramter for messages that may be originating from your third-party library. You may have to log all the WM_CREATE messages and then analyze the data to see how you can differentiate your library's dialog from any other dialog on the system. (Many times the "lpszName" member differs between dialog implementations.)
3. if you believe the particular WM_CREATE came from your library, return a "handled" response from your message hook and don't call the next message handler in the chain.

This is all very dicey mind you, but it is the path that seems like it has any chance to me.

Answer 3

Check out Detours from Microsoft Research. It allows you to detour arbitrary Windows API functions. C/C++ programming is required to make it work though. You won't need much.