HttpWebRequest.GetResponse() hangs the second time it is called

Question

I'm trying to fetch a series of files via HTTP, using HttpWebRequest. The first request goes through fine, but the second time through the same code GetResponse() hangs and times out. WireShark shows that no HTTP traffic is being sent for the second request, so it would appear that it's an API problem.

After some investigation, I've discovered that it has to do with specifying the content-length: if I leave this out, then the code works fine.

My code is:

HttpWebRequest  httpWebRequest = ConfigureRequest();

using (WebResponse webResponse = httpWebRequest.GetResponse())
    // On the second iteration we never get beyond this line
{
    HttpWebResponse httpWebResponse = webResponse as HttpWebResponse;

    using (Stream webResponseStream = httpWebResponse.GetResponseStream())
    {
        if (webResponseStream != null)
        {
            // Read the stream
        }
    }

    statusCode = httpWebResponse.StatusCode;
    httpWebResponse.Close();
}

The symptoms seem very similar to this question and this question, but in both cases the advice given is to dispose of the WebResponse, which I'm already doing.

Edit In response to Gregory, here is ConfigureRequest():

private HttpWebRequest ConfigureRequest()
{
    string          sUrl            = CreateURL(bucket, key);
    HttpWebRequest  httpWebRequest  = WebRequest.Create(sUrl) as HttpWebRequest;

    httpWebRequest.AllowWriteStreamBuffering = false;
    httpWebRequest.AllowAutoRedirect = true;
    httpWebRequest.UserAgent = this.m_sUserAgent;
    httpWebRequest.Method = "GET";
    httpWebRequest.Timeout = this.m_iTimeout;

    // *** NB: This line was left out of my original posting, and turned out to be
    // crucial
    if (m_contentLength > 0)
        httpWebRequest.ContentLength = m_contentLength;

    httpWebRequest.Headers.Add(StaticValues.Amazon_AlternativeDateHeader, timestamp);
    httpWebRequest.Headers.Add(StaticValues.HttpRequestHeader_Authorization, StaticValues.Amazon_AWS + " " + aWSAccessKeyId + ":" + signature);

    return httpWebRequest;
}

Edit: It turns out I committed the cardinal sin of removing code from my question that I hadn't verified was unrelated to the problem. I had removed the following lines:

    if (m_contentLength > 0)
        httpWebRequest.ContentLength = m_contentLength;

because I thought that content-length would never be specified for a GET request. It turns out I was wrong. Removing this line fixes the problem.

The only question I now have is why? I think the content length that is specified is correct, though it's possible it's off by one. Would specifying a content length that is too short prevent the full download from taking place and cause the connection to be left open? I would have expected that Close() and / or Dispose() ought to kill the connection anyway.

Answer 1

Make sure you are creating a NEW HttpWebRequest each time. Quote from the MSDN reference on GetResponse method :

Multiple calls to GetResponse return the same response object; the request is not reissued.

Update 1: Ok. How about if you try closing the webResponseStream too - you currently don't do that, you're only closing the webResponse (just trying to rule things out). I'd also dispose of the httpWebResponse too (not just the WebResponse)

Update 2: The only other thing I can suggest is to have a look at the following articles that I looked at when I did something similar to what you're doing:
http://arnosoftwaredev.blogspot.com/2006/09/net-20-httpwebrequestkeepalive-and.html
http://www.devnewsgroups.net/dotnetframework/t10805-exception-with-httpwebrequest-getresponse.aspx

The only noticeable things I do different are:
- set webrequest.KeepAlive = false
- I don't have the connection management stuff in the config (I don't loop round to make a series of requests)