This is not a complete answer, but may give you direction. If you are using IIS 7, then HttpModules are now native to IIS. That is, you can get an implementation of the IHttpModule interface to be used by IIS, regardless of what type of content is being served.
I suspect that you can wire such an HttpModule so that it will notice when IIS is failing a request due to a failure of Windows Authentication. If that is possible, then you might still be able to examine the POST variables and give the request a "second chance".
I also wonder if it might be possible with IIS 7 to turn this around - have the main site use anonymous authentication so that your code can check the form POST parameters. If that authentication fails, then redirect to the Windows Authentication site.