HttpWebRequest takes a long time to send when there are a bunch at once from client

Question

I am attempting to load-test a Comet-ish server using a C# load testing client that creates many HttpWebRequests (1000+). I am finding that after a few minutes, randomly, the server takes a long time to receive some of the requests. The client thinks it sent the request successfully, but it actually takes 40s to arrive at the server, at which point it is too late. (Since this is a Comet-type server, the server ends up dropping the client session, which is bad). I tried switching from asynchronous calls to synchronous calls but it didn't make a difference.

The problem must be at the client end. I did some tracing with Wireshark and it turns out that the request actually does take 40 or so seconds to make it to the network pipe from the client software! The server services the request right away when it receives it on its pipe.

Maybe C# is messing up because the request looks exactly the same as a request I made earlier and caching it for some weird reason? I am including "Cache-Control:no-cache" in my responses to avoid caching altogether.

Answer 1

I ran into a similar issue when I was first building my web crawler, which makes upwards of 2,000 requests every minute. The problem turned out to be that I wasn't always disposing of the HttpWebResponse objects in a timely fashion. The garbage collection / finalization mechanism will not keep up when you're making requests at that rate.

Whether you're doing synchronous or asynchronous requests doesn't really matter. Just make sure you always call response.Close().

Answer 2

You may be hitting the default client connection limitation. By default, it's two connections, and any more queue up behind it.

To get around this, add this to your app.config file:

  <system.net>
    <connectionManagement>
      <remove address="*"/>
      <add address="*" maxconnection="10" />
    </connectionManagement>
  </system.net>

Experiment with maxconnection to see where your effective upper limit is.