Why does HttpWebRequest GetResponse block for such a long time?

Question

I have some fairly simple code that uploads a photo or video to an endpoint (using HTTP PUT or POST). Every so often I see connection closed exceptions thrown, and in reality the photo/video was uploaded just fine, it's calling GetResponse where the exception occurs.

One thing I've noticed is that GetResponse can take an awful long time to process. Often longer than the actual upload time of the photo to the server. My code writes to the web server using RequestStream.Write.

I did a little test and uploaded about 40 photos/videos to the server that range in size from 1MB to 85MB and the time for GetResponse to return was anywhere from 3 to 40 seconds.

My question is, is this normal? Is this just a matter of how long the server I am uploading these files to is taking to process my request and respond? In looking at Fidder HTTP traces it seems to be the case.

FYI, my uploads are HTTP 1.0, Timeout values set to Infinite (both Timeout and ReadWriteTimeout)

Answer 1

Yes, the response time may be a lot longer than just the upload time. After the request has been sent to the server it has to be processed and a response has to be returned. There may be some time before the request is processed, and then the file typically is going to be saved somewhere. After that the server will create the response page that is sent back.

IIS handles only one request at a time from each user, so if you start another upload before the first one is completed, it will wait until the first one completes before it even starts to process the next.

Answer 2

If the server is genuinely taking a long time to return any data (as shown in Fiddler) then that's the cause of it. Uploading an 85MB attachment would take a long time to start with, and then the server has to process it. You can't do a lot about that - other than to use an asynchronous method if you're able to get on with more work before the call returns.

It's not entirely clear what Fiddler's showing you though - is it showing a long time before the server sends the response? If so, there's not much you can do. I'm surprised that the connection is being closed on you, admittedly. If, however, you're not seeing your data being written to the server for a while, that's a different matter.

Are you disposing the response returned? If not, you may have connections which are being kept alive. This shouldn't be a problem if it's explicitly HTTP 1.0, but it's the most common cause of "hanging" web calls in my experience.

Basically, if you don't dispose of a WebResponse it will usually (at least with HTTP 1.1 and keepalive) hold on to the connection. There's a limit to the number of connections which can be open to a single host, so you could end up waiting until an earlier response is finalized before the next one can proceed.

If this is the problem, a simple using statement is the answer:

using (WebResponse response = request.GetResponse())
{
    ...
}