Something I've done in the past is manually show/hide the modal popup. I realize the ModalPopupExtender control requires a target, so you'll need a dummy target which will remain inactive:
<asp:LinkButton id="btnDummyTarget" runat="server" />
<asp:Button
id="btnActualButtonTiedToValidation"
ValidationGroup="SomeValidationGroup"
OnClick="MyButton_Click"
runat="server" />
<ajaxToolkit:ModalPopupExtender
id="mpeMyPopup"
PopupControlID="pnlSomePanelToShow"
TargetControlID="btnDummyTarget"
runat="server" />
Then, in your codebehind you can do the following:
protected void MyButton_Click(object sender, EventArgs e)
{
if(Page.IsValid)
mpeMyPopupExtender.Show();
}
This is also handy for delete confirmation dialogs.