For some reason I have to face same problem in .NET.
I have (main) form showing modal dialog, which I need to hide, interact with main window, and return to the modal dialog again.
I personally do not understand consequences from the Windows (API) point of view, but following solution works for me.
Whole trick lies in setting main form to disabled before showing modal dialog (when main form is not set to Enabled = false explicitly, then after hiding modal dialog no interaction can be done with it even when Enabled = true is called).
As response to modal dialog event (called NeedInteraction), I just hide modal dialog, enable main form, in some loop do interaction with user, disable main dialog and show modal dialog again.
void ShowDialog()
{
var dialog = new MyModalForm();
dialog.NeedInteraction += (sender, eventArgs) =>
{
dialog.Hide();
Enabled = true;
//wait till user finishes working with main window
Enabled = false;
dialog.Show();
}
Enabled = false;
dialog.ShowDialog();
Enabled = true; //don't forget to make it enabled afterwards
}
It might not be clean solution (as is not need for hiding modal dialog), but it works at least for my situation.