Why can my property sheet, shown from a system tray icon, lock up the taskbar?

Question

Note: code samples have been simplified, but the overall structure remains intact.

I am working on a Win32 application whose main interface is a system tray icon. I create a dummy window, using HWND_MESSAGE as its parent, to receive the icon's messages:

WNDCLASSEX wndClass;
wndClass.lpfnWndProc = &iconWindowProc;
// ...
iconWindowHandle = CreateWindow(wndClass.lpszClassName, _T(""), 0, CW_USEDEFAULT, CW_USEDEFAULT, CW_USEDEFAULT, CW_USEDEFAULT, HWND_MESSAGE, NULL, GetModuleHandle(NULL), 0);

Then the icon is created, referring to this message-only window:

NOTIFYICONDATA iconData;
iconData.hWnd = iconWindowHandle;
iconData.uCallbackMessage = TRAYICON_MESSAGE;
// ...
Shell_NotifyIcon(NIM_ADD, &iconData)

When the tray icon is double-clicked, I create and show a property sheet (from comctl32.dll):

LRESULT CALLBACK iconWindowProc(HWND hWnd, UINT uMsg, WPARAM wParam, LPARAM lParam) {
  switch (uMsg) {
    case TRAYICON_MESSAGE:
      switch (lParam) { // that contains the "real" message
        case WM_LBUTTONDBLCLK:
          showPropertySheet();
          return 0;
        // ...
      }
      break;
    // ...
  }
  return DefWindowProc(hWnd, uMsg, wParam, lParam);
}

The property sheet has no parent window. The PropertySheet function is called from the window procedure of the message-only window. The PSH_MODELESS flag is not set; thus, PropertySheet only returns after the property sheet window is closed again:

void showPropertySheet() {
  PROPSHEETPAGE pages[NUM_PAGES];
  pages[0].pfnDlgProc = &firstPageDialogProc;
  // ...
  PROPSHEETHEADER header;
  header.hwndParent = NULL;
  header.dwFlags = PSH_PROPSHEETPAGE | PSH_USECALLBACK;
  header.ppsp = pages;
  // ...
  PropertySheet(&header);
}

Now all this works just fine, until I set a breakpoint inside the dialog procedure of one of the property sheet's pages:

BOOL CALLBACK firstPageDialogProc(HWND hWnd, UINT uMsg, WPARAM wParam, LPARAM lParam) {
  return FALSE; // breakpoint here
}

When the program stops on the breakpoint, the entire taskbar locks up!

The call stack is quite useless; it shows that the dialog procedure is called from somewhere inside comctl32.dll, via some calls inside user32.dll. No window procedure of my own is in between.

Making the property sheet modeless doesn't seem to help. Also, I'd rather not do this because it makes the code more complex.

As long as my dialog procedure returns quickly enough, this shouldn't be a problem. But it seems so weird that a longer operation inside the dialog procedure would not only lock up the dialog itself, but the entire shell. I can imagine that the message-only window procedure has the power to cause this behaviour, since it's more closely related to the tray icon... but this function is not shown on the call stack.

Am I doing something fundamentally wrong? Can anyone shed some light on this issue?

Answer 1

Actually, it's rather obvious, and the confusion must have been due to a lack of coffee.

The taskbar probably uses SendMessage to send the message to my application, which causes it to block until the message is handled. SendMessageTimeout is apparently not used.

I still think it's strange that no function of my own shows up on the call stack. Surely, such a message must flow through my message loop in order to be processed? Maybe the warning that "stack frames below this line may be incomplete or missing" was actually right, then.