PInvokeStackImbalance - invoking unmanaged code from HookCallback

Question

my goal

i want to translate a left click to a right click

my approach

• i register a low-level hook via SetWindowsHookEx (user32.dll)
• filter left-mouse-clicks
• check if i want to translate THAT specific click
• in case that i really want to
  • do not pass on the message
  • create a new mouseclick via mouse_event (user32.dll too)

the problem

when i do the described stuff like that:

  private static IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam) {
     if(nCode >= 0 && MouseMessages.WM_LBUTTONDOWN == (MouseMessages)wParam && doRight) {
        doRight = false;
        MSLLHOOKSTRUCT hookStruct = (MSLLHOOKSTRUCT)Marshal.PtrToStructure(lParam, typeof(MSLLHOOKSTRUCT));
        mouse_event(/*right down + right up*/8 | 16, hookStruct.pt.x, hookStruct.pt.y, 0, 0);
        return new IntPtr(1);
     }
     return CallNextHookEx(_hookID, nCode, wParam, lParam);
  }

the call to mouse_event fails with a PInvokeStackImbalance-Exception, which, i guess, i should care about.

DllImports

since normally a PInvokeStackImbalance comes due to incorrect import-signatures here are mine:

  [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
  private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelMouseProc lpfn, IntPtr hMod, uint dwThreadId);

  [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
  [return: MarshalAs(UnmanagedType.Bool)]
  private static extern bool UnhookWindowsHookEx(IntPtr hhk);

  [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
  private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode, IntPtr wParam, IntPtr lParam);

  [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
  private static extern IntPtr GetModuleHandle(string lpModuleName);

  [DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
  public static extern void mouse_event(long dwFlags, long dx, long dy, long cButtons, long dwExtraInfo);

emergence

my normal approach to isolate the problem fails in that case - since the mouse_event-call for itself works, and the trashing of left-clicks work too. i hate it when the structure is more than the sum of the parts...

Answer 1

The mouse_event signature should be like this

public static extern void mouse_event(int dwFlags, int dx, int dy, int cButtons, IntPtr dwExtraInfo);

Answer 2

mouse_event is defined as:

VOID WINAPI mouse_event(
  __in  DWORD dwFlags,
  __in  DWORD dx,
  __in  DWORD dy,
  __in  DWORD dwData,
  __in  ULONG_PTR dwExtraInfo
);

WINAPI means StdCall, DWORD is uint, ULONG_PTR is UIntPtr. So correctly it should be:

[DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
public static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint cButtons, UIntPtr dwExtraInfo);

The problem is that long is defined in C# as 64-bit, while (u)int is 32-bit. (U)IntPtr is 32-bit or 64-bit, depending on the bitness of the operating system.

EDIT: In other words, you are passing too much data to the function. Since the callee cleans the stack, PInvoke notices that not everything was removed from the stack. It does it for you (but you have passed the wrong data, so the function has likely done something different from what you wanted) and warns you.

Answer 3

There are a lot of bad p/invoke declarations on the Internet. They typically started life in VB6. In that language, Integer is 16 bits and Long is 32 bits. Goes back to VB1 which ran on 16-bit operating systems. They don't work properly in VB.NET or C#.

The stack imbalance MDA was specifically designed to catch such bad declarations, but it is an option (Debug + Exceptions). The actual call tends to work if there is only one argument or when passing a lot of zeros. The odds that code keeps running properly after the call is, unfortunately, not bad either. The stack pointer value corrects itself when the method returns. You can get some really screwy problems though.