.NET - download a web page with multiple authentication

Question

There is a particular secure web page that we are required to download and parse. Its from this parsed data we need to do a host of other things... The so-called 'multiple authentication' part goes like the follows:

If you've hosted web sites on IIS and turned on the Integrated Security feature (with anonymous browsing turned off), you'd notice that the instant you type in the url to that page a window pops out of your browser asking you for your windows network credentials...this particular external web page has something similar to that.

The weird thing is that it shows that window twice. And the page only loads correctly when I provide my credentials twice. Yes, I have to provide my credentials twice!!! I have not come across such a scenario. I did notice the title of the pop up and figured that that is the url where I've to provide my credentials.

To download web pages I make use of the HttpWebRequest & HttpWebResponse objects. For the web request object I have to set the credentials appropriately. For this I had created a CredentialsCache object and added the two credentials (Ref http://msdn.microsoft.com/en-us/library/system.net.credentialcache.aspx for sample code). Still no luck.

Has anyone come across this scenario?

Answer 1

The weird thing is that it shows that window twice. And the page only loads correctly when I provide my credentials twice. Yes, I have to provide my credentials twice!!!

From the browser side of things this sounds like a simple issue with the html. I've seen this often enough on secure sites that are inconsistent about fully-qualified URLs. For instance, if you visit https://mywebsite.com/default.html, and in that html it has a picture link like <img src="http://MyWebsite.com/picture.jpg">, then your going to get two prompts. The reason in this case is because of the case-sensitivity of the URL, but it could be that it uses a completely different name like 'www.mywebsite.com' instead.

Another possibility is a server-side redirect to another secure url. You can find out if this is happening by setting the HttpWebRequest.AllowAutoRedirect to false, and then checking the response headers for a redirect.

Just an idea on what might be causing this? This is likely going to very specific to your environment so It's kind of hard to help much.

Answer 2

To add to what csharptest noted, I'd look at the traffic in Fiddler (or whatever you like for http debugging) to try and see what is going on.

Answer 3

You can enable trace logging using http://ferozedaud.blogspot.com/2009/08/tracing-with-systemnet.html and see the log. It will tell you what authentication is happening.

Even if you are getting multiple prompts from different domains (due to a redirect for eg) you can still make that succeed by setting credentials appropriately in the CredentialCache for those two sites/domains.