I noticed that you have implemented all of the interoperability yourself. Can you try this with the GeckoFX project and see if you get the same error? I use this project at work and haven't encountered any issues yet.
The WM_CHAR message is generated by TranslateMessage call, so a good place to start looking would be the TranslateMessage calls in the Gecko source code.
In the first example code you provided the function is imported only by two libraries - mozctl.dll and xul.dll. Since you claim that the same error happens also with GeckoFX we can take mozctl.dll out of the equation. That leaves us with xul.dll, so given the Gecko source code I would suggest to look into widget\src\windows\nsToolkit.cpp. I am not sure if the code is run if the engine is embedded, but if it is then the library starts a whole new message pump in different thread, which is bound to break.
Unfortunately I can't run or compile the code on my machine (Windows 7 x64 w/o the Mozilla ActiveX control installed), so I can't verify any of this with a debugger. Hope it helps someone to track it down further.
@OP: Did you resolve the problem? Wouldn't it better to inject javascript in the automated page than sending keys?
The root problem is that when Mozilla is embedded in another application, it incorrectly pumps Windows messages when it dispatches internal events. Mozilla uses an event system to coordinate across threads or to schedule deferred processing on a thread (see nsIThread, nsIEventTarget). If you embed a web page with a lot of active XMLHTTPRequests, for example, Mozilla will use its event dispatching interface to dispatch events back to javascript and it will pump windows messages as a side effect. Once Mozilla events are fully dispatched, it goes back to the main event loop.
When Mozilla pumps windows messages, it doesn't include the extra processing done by the application's event loop - IsDialogMessage(), TranslateMessage(), PreTranslateMessage(), or any other pre-processing are skipped when Mozilla gets into this state. Symptoms therefore include tab key presses getting inserted as a character instead of being used for dialog navigation, keyboard hotkeys being sporadically ignored, or custom message pre-processing being sporadically skipped. For example, the Outlook 2007/2010 "Compose" screen sporadically loses keystrokes because it relies on custom message pre-processing to handle keyboard input.
See https://bugzilla.mozilla.org/show_bug.cgi?id=582790 for a patch that addresses the problem.
I have Snoop Free and PSM Anti-Keylogger. One of them detected firefox trying to install a Keyboard Hook. Mozilla/Firefox file xul.dll attempt at installing at keyboard hook. DENIED.