Can I suppress selected input before the application's main loop?

Question

As part of my Visual Studio utilities add-in SamTools, I have a mouse input routine that catches Ctrl+MouseWheel and sends a pageup/pagedown command to the active text window. Visual Studio 2010 added a new "feature" that uses that gesture for zoom in/out (barf). Currently, my add-in does send the scrolling command, but Visual Studio still changes the font size because I'm not eating the input.

I set my hook with a call to SetWindowsHookEx. Here's the callback code. My question is: is the best way to prevent Visual Studio from handling the Ctrl+MouseWheel input as a zoom command to simply not call CallNextHookEx when I get a mouse wheel event with the Ctrl key down?

(Please bear in mind this is some old code of mine.) :)

private IntPtr MouseCallback(int code, UIntPtr wParam, ref MOUSEHOOKSTRUCTEX lParam)
{
    try
    {
        // the callback runs twice for each action - this is the latch
        if (enterHook)
        {
            enterHook = false;
            if (code >= 0)
            {
                int x = lParam.mstruct.pt.X;
                int y = lParam.mstruct.pt.Y;

                uint action = wParam.ToUInt32();
                switch (action)
                {
                case WM_MOUSEWHEEL:
                    OnMouseWheel(new MouseEventArgs(MouseButtons.None, 0, x, y, ((short)HIWORD(lParam.mouseData)) / (int)WHEEL_DELTA));
                    break;

                default:
                    // don't do anything special
                    break;
                }
            }
        }
        else
        {
            enterHook = true;
        }
    }
    catch
    {
        // can't let an exception get through or VS will crash
    }

    return CallNextHookEx(mouseHandle, code, wParam, ref lParam);
}

And here's the code that executes in response to the MouseWheel event:

void mouse_enhancer_MouseWheel( object sender, System.Windows.Forms.MouseEventArgs e )
{
    try
    {
        if ( Keyboard.GetKeyState( System.Windows.Forms.Keys.ControlKey ).IsDown && Connect.ApplicationObject.ActiveWindow.Type == vsWindowType.vsWindowTypeDocument )
        {
            int clicks = e.Delta;
            if (e.Delta < 0)
            {
                Connect.ApplicationObject.ExecuteCommand( "Edit.ScrollPageDown", "" );
            }
            else
            {
                Connect.ApplicationObject.ExecuteCommand( "Edit.ScrollPageUp", "" );
            }
        }
    }
    catch ( System.Runtime.InteropServices.COMException )
    {
        // this occurs if ctrl+wheel is activated on a drop-down list. just ignore it.
    }
}

PS: SamTools is open source (GPL) - you can download it from the link and the source is in the installer.

PSS: Ctrl+[+] and Ctrl+[-] are better for zooming. Let Ctrl+MouseWheel scroll (the vastly more commonly used command).

Answer 1

According to MSDN, it's possible to toss mouse messages that you process. Here's the recommendation:

If nCode is less than zero, the hook procedure must return the value returned by CallNextHookEx.

If nCode is greater than or equal to zero, and the hook procedure did not process the message, it is highly recommended that you call CallNextHookEx and return the value it returns; otherwise, other applications that have installed WH_MOUSE hooks will not receive hook notifications and may behave incorrectly as a result. If the hook procedure processed the message, it may return a nonzero value to prevent the system from passing the message to the target window procedure.

In other words, if your mouse callback ends up using the mouse message, you don't have to call the next CallNextHookEx -- just return a nonzero value and (in theory, at least) the mouse movement should get swallowed. If that doesn't work the way you want, comment and we can iterate.

BTW, another possible alternative: it's possible that VS's mapping to the mouse wheel shows up in the Tools...Customize... UI, just like key mappings do. In that case, you could simply remap your add-in's commands instead of working at the hook level. But it's also posible (likely?) that this gesture is hard-coded.