HttpWebRequest is extremely slow!

Question

Hello, I am using an open source library to connect to my webserver. I was concerned that the webserver was going extremely slow and then I tried doing a simple test in Ruby and I got these results

Ruby program: 2.11seconds for 10 HTTP GETs

Ruby program: 18.13seconds for 100 HTTP GETs

C# library: 20.81seconds for 10 HTTP GETs

C# library: 36847.46seconds for 100 HTTP GETs

I have profiled and found the problem to be this function:

private HttpWebResponse GetRawResponse(HttpWebRequest request) {
  HttpWebResponse raw = null;
  try {
    raw = (HttpWebResponse)request.GetResponse(); //This line!
  }
  catch (WebException ex) {
    if (ex.Response is HttpWebResponse) {
      raw = ex.Response as HttpWebResponse;
    }
  }
  return raw;
}

The marked line is takes over 1 second to complete by itself while the ruby program making 1 request takes .3 seconds. I am also doing all of these tests on 127.0.0.1, so network bandwidth is not an issue.

What could be causing this huge slow down?

UPDATE

Check out the changed benchmark results. I actually tested with 10 GETs and not 100, I updated the results.

Answer 1

Try switching to sockets HttpWebRequest is a layer of abstraction over sockets. To get better performance, go down a layer of abstraction:

http://www.codeproject.com/KB/IP/Crawler.aspx#WebRequest

Answer 2

It may have to do with the fact that you are opening several connections at once. By default the Maximum amount of open HTTP connections is set to two. Try adding this to your .config file and see if it helps:

<system.net>
  .......
  <connectionManagement>
    <add address="*" maxconnection="20"/>
  </connectionManagement>
</system.net>

Answer 3

Use a computer other than localhost, then use WireShark to see what's really going over the wire.

Like others have said, it can be a number of things. Looking at things on the TCP level should give a clear picture.

Answer 4

What I have found to be the main culprit with slow web requests is the proxy property. If you set this property to null before you call the GetResponse method the query will skip the proxy autodetect step.

The proxy autodetect was taking up to 7 seconds to query before returning the response. It is a little annoying that this property is set on by default for the HttpWebRequest object.