Interfere Win32 message loop with injected DLL code (SetWindowsHookEx)

Question

Hello everybody!

After hours of penetrating Google I ended up here. I'll come straight to the point: I'm about to "refresh" my C/C++ skills and gain experience with the unmanaged world again. As a "basic" task I developed a little key logger (which are just a few lines with the Windows API) but now I want to extend it with a "stealth" feature. Therefor I threw the code into a Win32 DLL it's content you find here. As you will notice, there is a very problematic part in it:

  MSG msg;
 BOOL bRet;

 while( (bRet = GetMessage( &msg, NULL, 0, 0 )) != 0)
 { 
  if (bRet == -1)
  {
   return FALSE;
  }
  else
  {
   TranslateMessage(&msg); 
   DispatchMessage(&msg); 
  }
 }

Yes, it's an ordinary message loop - which causes trouble in my case. I inject this DLL into a "victim" executable (e.g. VLC media player) to fool AV/Application firewalls and it works so far, the injection itself passes flawlessly. Now comes the big BUT: of course the endless while-loop now freezes the whole target application (without it, my hook callback never gets executed) which wasn't really planed... After diving through half of the MSDN library and trying a lot of "solutions" Google gave me; I give up.

Is it even possible to evaluate the message loop of the "victim" process without blocking it's own business but providing my keyboard hook callback to work?

Sincerely yours, Nefarius

Answer 1

You shouldn't have a message loop there. The application you're injecting into already has a message loop (unless it's a console app, which doesn't deal with messages anyway). Just let your hook do its thing when the host's message loop processes its messages as it normally would.

Answer 2

Okay, first off, you're doing way too much in your dll entry point function. For one thing - and this is straight from MSDN - "There are serious limits on what you can do in a DLL entry point". Also, while in the dll entry point the loader lock is held so no other libraries can be loaded/unloaded. So seeing as you're running your message loop (by calling InstallHook()) in the DLL entry point, you're really throwing a stick in the bicycle spokes, so the speak.

Now with that out of the way, getting it to work is pretty simple. When the DLL is loaded, create a new thread at InstallHook and you should be good to go. Now you're message loop will be in it's own thread with it's own message queue (or at least it should, windows messaging still kinda confuses me).

case DLL_PROCESS_ATTACH:
  CreateThread( NULL, 0, (LPTHREAD_START_ROUTINE)InstallHook, (void*)NULL, 0, NULL );