HTTP 407 proxy authentication error when calling a web service

Question

I'm working on a .NET app that calls 3rd party web services over the internet. The services do not use SOAP, so we manually construct an XML request document, send it to the service via HTTP, and retrieve an XML response.

Our code is a Windows service that is run in the context of a normal Windows domain account, and sits behind a proxy server (Microsoft ISA Server) configured to require NTLM authentication. The account running our service has permission to access the internet through the proxy server.

The code looks like this:

// Create the request object.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";

// Configure for authenticating proxy server requiring Windows domain credentials.
request.Proxy = New WebProxy(proxyAddress) { UseDefaultCredentials = true };

// Set other required headers.
request.Accept = acceptableMimeType;
request.Headers.Add(HttpRequestHeader.AcceptCharset, acceptableCharset);
request.Headers.Add(HttpRequestHeader.AcceptEncoding, "none");
request.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-gb");
request.Headers.Add(HttpRequestHeader.CacheControl, "no-store");
request.Headers.Add(HttpRequestHeader.ContentEncoding, "none");
request.Headers.Add(HttpRequestHeader.ContentLanguage, "en-gb");
request.ContentType = requestMimeType;
request.ContentLength = requestBytes.Length;

// Make the method call.
using(Stream stream = request.GetRequestStream()) {
    stream.Write(requestBytes, 0, requestBytes.Length);
}
HttpWebResponse response = (HttpWebResponse) request.GetResponse();

// Extract the data from the response without relying on the HTTP Content-Length header
// (we cannot trust all providers to set it correctly).
const int bufferSize = 1024 * 64;
List<byte> responseBytes = new List<byte>();
using(Stream stream = new BufferedStream(response.GetResponseStream(), bufferSize)) {
    int value;
    while((value = stream.ReadByte()) != -1) {
        responseBytes.Add((byte) value);
    }
}

This works fine if the proxy server is turned off, or the URL has been whitelisted as not requiring authentication, but as soon as authentication is active, it always fails with an HTTP 407 error.

I put the above code in a test harness, and tried every method I could think of for configuring the request.Proxy property, without success.

I then noticed that all the 3rd party web services that we have to call are HTTPS. When I tried accessing them as HTTP instead, the proxy authentication started working. Is there some extra hoop I have to jump through to get proxy authentication and HTTPS to play nicely?

PS: The same problems occur with the open source SmoothWall proxy server, so I can't just write it off as a bug in ISA Server.

PPS: I'm aware that you can configure proxy settings in app.config, but (a) doing it in code shouldn't make any difference, and (b) the application design requires that we read the proxy settings from a database at runtime.

Answer 1

Have you tried setting the proxy in the app.config ?

To disable the proxy, in the App.config file add the following configuration

<system.net>
  <defaultProxy enabled="false" useDefaultCredentials="false">
    <proxy/>
    <bypasslist/>
    <module/>
  </defaultProxy>
</system.net>

To enable the proxy and to use the default proxy settings(specified in IE) add this configuration in your App.config

<system.net>
  <defaultProxy enabled="true" useDefaultCredentials="true">
    <proxy/>
    <bypasslist/>
    <module/>
  </defaultProxy>
</system.net>

Answer 2

Is there something wrong with your proxy server's certificate? If your service can't establish HTTPS then it will throw an error.

Answer 3

I think I will have to write off this question. My original posted code does appear to work sometimes. Our proxy server is extremely unreliable; one minute it will block an internet connection from any software, and the next it will allow it. The IT guys seem powerless to do anything about it, and we (everyone outside the IT department) have no authority to make changes to the network infrastructure.

If anyone has any ideas on how to "harden" my code to compensate for an unreliable proxy server, then I'd be interested to hear them. :-)

Answer 4

I did have a similar situation

Did you noticed it worked when you accessed the internet before you ran the code? and if you had not accessed the internet for ages (20mins for me) you got the error.

have you tried to set the proxy credentials directly?

//setup the proxy
request.Proxy = new WebProxy("proxyIp", 8080);
request.Proxy.Credentials = CredentialCache.DefaultCredentials;

I recently blogged about this.

I hope this fixes your issue too!