[Named Pipes] Application to Global Hook DLLs

Question

My objective is conceptually simple: I want to set a GetMessage global hook function that utilizes a shared file handle. The problem arises because to my understanding the DLL containing the hook function is loaded multiple times for each process, each with their own "address space". For this reason I'm led to believe I can't simply handle DllMain's DLL_PROCESS_ATTACH to create the desired file, as multiple files would be created with different handles.

A solution that's been brought to my attention is Named Pipes. Basically the application would act as the server end; it would create the file one time then provide the file handle to the DLL clients, therefore each global hook would be using the same file.

I can't seem to get it to work from the code I gathered. In the application, I create the file, set the global hook function, then make it go through this loop:

while(1)
{
  HANDLE hPipe = CreateNamedPipe("\\\\.\\pipe\\pipename", PIPE_ACCESS_OUTBOUND, 
PIPE_TYPE_BYTE|PIPE_READMODE_BYTE|PIPE_WAIT, PIPE_UNLIMITED_INSTANCES, 32, 32, 5000, NULL);
  if(hPipe == INVALID_HANDLE_VALUE)
    return 42;
  if(!ConnectNamedPipe(hPipe, NULL))
    return 43;
  DWORD dwWritten;
  WriteFile(hPipe, logFile, sizeof(logFile), &dwWritten, NULL);
  FlushFileBuffers(hPipe);
  DisconnectNamedPipe(hPipe);
  CloseHandle(hPipe);
}

Then I handle the DllMain's DLL_PROCESS_ATTACH as so:

case DLL_PROCESS_ATTACH:
{
  HANDLE hPipe;
  while(1)
  {
    hPipe = CreateFile("\\\\.\\pipe\\pipename", GENERIC_READ, 0, NULL, OPEN_EXISTING, 0, NULL);
    if(hPipe != INVALID_HANDLE_VALUE)
      break;
    WaitNamedPipe("\\\\.\\pipe\\pipename", NMPWAIT_USE_DEFAULT_WAIT);
  }
  DWORD dwRead;
  ReadFile(hPipe, logFile, sizeof(logFile), &dwRead, NULL);
  CloseHandle(hPipe);
  break;
}

Simply put, it does not work, and I can't seem to figure out why. Is there something I'm missing or doing wrong in my code?

Another problem that I can't figure out is that the application is stuck in an infinite loop of constantly serving. I want to set up an Event that the DLL will set under a certain circumstance and cause the main application to unhook the global hook, close the file and exit, however ConnectNamedPipe is a blocking function. What is a way to determine when all clients have been served so the serving loop can break?

Thanks for any help.

Answer 1

There are strong restrictions as to what system APIs you may call during DLL_PROCESS_ATTACH or DLL_THREAD_ATTACH. From the MSDN documentation.

The entry-point function should perform only simple initialization or termination tasks. It must not call the LoadLibrary or LoadLibraryEx function (or a function that calls these functions), because this may create dependency loops in the DLL load order. This can result in a DLL being used before the system has executed its initialization code. Similarly, the entry-point function must not call the FreeLibrary function (or a function that calls FreeLibrary) during process termination, because this can result in a DLL being used after the system has executed its termination code.

Because Kernel32.dll is guaranteed to be loaded in the process address space when the entry-point function is called, calling functions in Kernel32.dll does not result in the DLL being used before its initialization code has been executed. Therefore, the entry-point function can call functions in Kernel32.dll that do not load other DLLs. For example, DllMain can create synchronization objects such as critical sections and mutexes, and use TLS. Unfortunately, there is not a comprehensive list of safe functions in Kernel32.dll.

Windows 2000: Do not create a named synchronization object in DllMain because the system will then load an additional DLL.

Calling functions that require DLLs other than Kernel32.dll may result in problems that are difficult to diagnose. For example, calling User, Shell, and COM functions can cause access violation errors, because some functions load other system components. Conversely, calling functions such as these during termination can cause access violation errors because the corresponding component may already have been unloaded or uninitialized.

For experiment-class work, consider using the thread attach event to see "what happens." For production work, you will need a completely revised approach which does no heavy work within DllMain. You can see above that future history will include more bugs in this OS facility.

Answer 2

It seems to me that your main problem could be the last parameter of the CreateNamedPipe function (SECURITY_ATTRIBUTES) or other security problems (see below).

I don't really understood which kind of information you plan to wrire in logFile which can be not more as 32 bytes (16 WCHARs). Usage of sizeof() in CreateNamedPipe would be also a little better (think also about 64-bit operation systems). Do you want to send a handle to a log-file opened in one process to other processes? If you do this you should use functions like DuplicateHandle (see http://msdn.microsoft.com/en-us/library/ms724251.aspx). In general, the code example for communication with respect of the named pipe which you posted I find not quite good. I recommend you first of all to debug the named pipe communication oufside of the hooking DLL (with at lest two separate client processes, which better runs under different user credentials and a server process which created the pipe).

There are of cause restriction in the usage of Windows API inside of DLL_THREAD_ATTACH, but in the usage of Kernel32.dll in your case seems me safe.

I don't know what kind of communication you want to implement, but in general the usage of unblocked mode like the usage of completion routines or other asynchronous operation (see http://msdn.microsoft.com/en-us/library/aa365788.aspxhttp://msdn.microsoft.com/en-us/library/aa365788.aspx and http://msdn.microsoft.com/en-us/library/aa365601.aspx) inside of DllMain could be better.

One more small recommendation: you should use DisableThreadLibraryCalls() in the case DLL_PROCESS_ATTACH and try to choose a reasonable base address of the DLL (a linker switch) which will reduce DLL relocation during its loading in different processes. These will speed up a little you program and save memory.