407 Authentication required - no challenge sent

Question

Update:
If you've just arrived at this question, the general gist is that I'm trying to make a HttpWebRequest via a proxy, and I'm getting a 407 from our strange proxy server. IE,Firefox,Chrome all manage to negotiate the proxy sucessfully, as do Adobe Air applications. It may be important that the Google Chrome web installer actually fails and we have to use an offline installer.

Thanks to Ian's link I've got it getting through to the next stage. It is now sending a token back to the proxy, however the 3rd stage isn't getting through, so the request with the username/password hash isn't being sent by .NET and consequently no HTML is returned.

I am using:

• IE6 user-agent
• Windows 7
• Scansafe proxy
• .NET 3.5

Here's the latest code that equates to the logs below:

HttpWebRequest request = HttpWebRequest.Create("http://www.yahoo.com") as HttpWebRequest;
IWebProxy proxy = request.Proxy;
// Print the Proxy Url to the console.
if (proxy != null)
{
    // Use the default credentials of the logged on user.
    proxy.Credentials = CredentialCache.DefaultCredentials;
}
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";
request.Accept = "*/*";

HttpWebResponse response = request.GetResponse() as HttpWebResponse;
Stream stream = response.GetResponseStream();

The exception

WebException (407) Authentication Required.

The proxy being used

The proxy client is a Scansafe hardware device in our server room, which (once authenticated with NTLM) then directs your HTTP traffic to its servers to filter the traffic.

System.Net tracing output

• Here's the trace output

IE sucessfully negotiating the proxy

• Fidder output

The solution

I haven't really found a solution but thanks to Feroze and Eric I have found a workaround and discovered that the actual proxy (and not its configuration) is the main issue. It may be an obscure issue with 3 variables: .NET HttpWebRequest's implementation, Windows 7 and of course the Scansafe hardware client that sits in our rack; but without an MSDN support request I won't find out.

Answer 1

I was having a similar issue and used the tips in the following blog post to solve the problem:

http://blogs.msdn.com/jpsanders/archive/2009/03/24/httpwebrequest-webexcepton-the-remote-server-returned-an-error-407-proxy-authentication-required.aspx

Answer 2

Can you try setting the User-Agent header on your HttpWebRequest, to the same value that IE8 is setting?

Sometimes, servers will not challenge correctly if the user-agent is not what they expect.

hope this helps.

Answer 3

If you want to set credentials for the proxy, shouldn't you set credentials on the request.Proxy object rather than the request object?

http://msdn.microsoft.com/en-us/library/system.net.webproxy.credentials.aspx

Also, keep in mind that you need to be making a HTTP/1.1 request (or technically, any request with Keep-Alive) to successfully use NTLM/Negotiate authentication.

(Fiddler's "Auth" inspector will decompose the NTLM authentication blobs for you, if you haven't taken a look at that yet.)

Answer 4

Is it the way the proxy is assigned?

proxy.Credentials = CredentialCache.DefaultCredentials;

When I last used proxy with the HttpWebRequest it was assigned like this:

Assign proxy to request:

request.Proxy.Credentials = Credentials.GetProxyCredentials();

Calls method:

    public static ICredentials GetProxyCredentials()
    {
        return new NetworkCredential(AppConstants.Proxy_username, AppConstants.Proxy_password);
    }

Configure proxy in web.config

<system.net>
  <defaultProxy enabled="true">
    <proxy
      autoDetect="False"
      bypassonlocal="True"
      scriptLocation="http://www.proxy.pac"
      proxyaddress="http://proxy1.blah.com" />
  </defaultProxy>
</system.net>

Answer 5

It could be related to what is in your "CredentialCache". Try this instead:

proxy.Credentials = new NetworkCredential("username", "pwd", "domain");

Answer 6

How about this:

        HttpWebRequest request = HttpWebRequest.Create("http://www.yahoo.com") as HttpWebRequest;
        WebProxy proxyObject = new System.Net.WebProxy("http://10.0.0.1:8080/", true); //whatever your proxy address is

        proxyObject.Credentials = CredentialCache.DefaultCredentials;
        request.Proxy = proxyObject;

        request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";
        request.Accept = "*/*";

        HttpWebResponse response = request.GetResponse() as HttpWebResponse;
        Stream stream = response.GetResponseStream();

Answer 7

I wrote a utility to decode the NTLM blobs that were sent in the IE and HttpWebRequest sessions.

When I look at the HttpWebRequest and IE, they both request 56bit and 128bit encryption from the server. Here is the dump of the session using HttpWebRequest

==== Type1 ----
Signature: NTLMSSP
Type: 1
Flags: E20882B7
NTLMSSP_NEGOTIATE_56
NTLMSSP_NEGOTIATE_KEY_EXCH
NTLMSSP_NEGOTIATE_128
RESERVED2
RESERVED3
RESERVED4
NTLMSSP_REQUEST_NON_NT_SESSION_KEY
NTLMSSP_TARGET_TYPE_DOMAIN
NTLMSSP_NEGOTIATE_OEM_DOMAIN_SUPPLIED
NTLMSSP_NEGOTIATE_DATAGRAM
NTLMSSP_REQUEST_TARGET
NTLM_NEGOTIATE_OEM
NTLMSSP_NEGOTIATE_UNICODE)
Domain :
Workstation:
==== Type2 ----
Signature: NTLMSSP
Type: 2
Flags: 201
NTLMSSP_NEGOTIATE_56
NTLMSSP_REQUEST_NON_NT_SESSION_KEY)
Context: D32FDDCB:63507CFA

Here is the dump from IE:

==== Type1 ----
Signature: NTLMSSP
Type: 1
Flags: A208B207
NTLMSSP_NEGOTIATE_56
NTLMSSP_NEGOTIATE_KEY_EXCH
NTLMSSP_NEGOTIATE_128
NTLMSSP_REQUEST_NON_NT_SESSION_KEY
NTLMSSP_NEGOTIATE_EXTENDED_SESSIONSECURITY
NTLMSSP_TARGET_TYPE_SHARE
NTLMSSP_TARGET_TYPE_DOMAIN
NTLMSSP_NEGOTIATE_OEM_DOMAIN_SUPPLIED
NTLMSSP_NEGOTIATE_DATAGRAM
NTLMSSP_REQUEST_TARGET
NTLMSSP_NEGOTIATE_UNICODE)
Domain : XXXX.UK
Workstation: XXX-X31
==== Type2 ----
Signature: NTLMSSP
Type: 2
Flags: 201
NTLMSSP_NEGOTIATE_56
NTLMSSP_REQUEST_NON_NT_SESSION_KEY)
Context: D32FDDCB:63507CFA

In both IE/HttpWebRequest, they are requesting both 64 & 128bit security. However, for windows7, 128bit security for NTLM has been made the default, and without that, authentication will fail. As you can see from the server response, the server is only supporting 64bit encryption.

The following link has a discussion on a similar problem encountered by another person. http://social.msdn.microsoft.com/Forums/en-US/ncl/thread/f68e8878-53e9-4208-b589-9dbedf851198

The reason that IE works, instead of the managed app, is that IE does not actually request NTLMSSP_NEGOTIATE_SEAL | NTLMSSP_NEGOTIATE_SIGN, which end up requiring encryption. However, HttpWebRequest does request both SEAL|SIGN. This requires 128bit encryption, whereas the way IE initializes the NTLMSSP (without SEAL & SIGN), it does not require encryption. Hence IE works, whereas HttpWebRequest does not. (see the link above)

I think that if you change your security policy to allow 64bit encryption for NTLM, your managed code app will work. Or alternately, ask the proxy vendor to support 128bit encryption for NTLM.

Hope this helps.