Why does the performance of the HttpWebRequest object improve while using Fiddler?

Question

I'm getting some very strange behaviour with HttpWebRequest I hope someone can help me with. I have a console app which does some aggregation work by by using the HttpWebRequest object to retrieve the contents of a target website. Due to the nature of the requirement the app is multithreaded and attempts to make anywhere between 10 and 30 simultaneous connections (I've been experimenting with a range of values). The actual web request is structured as follows:

var req = (HttpWebRequest)WebRequest.Create(url);
WebResponse resp = req.GetResponse();
Stream s = resp.GetResponseStream();
var sr = new StreamReader(s, Encoding.ASCII);
string doc = sr.ReadToEnd();
sr.Close();
resp.Close();
return doc;

Anyway, the strange behaviour is that under normal circumstances the app is achieving around 120 requests per minute but if I open up Fiddler it jumps to about 600. Using Windows 7 Resource Monitor I can see the network activity increase accordingly. The TCP connections for the console process now list the remote address as "IPv4 loopback" rather than the target server IP address (expected). I did wonder about the max number of simultaneous HTTP requests allowed by the machine but changing this in the registry does not seem to make a difference.

So the question is; what is it about running Fiddler which suddenly increases the throughput five-fold and how can I achieve this natively on the machine without needing to launch another tool?

Thanks!

Answer 1

One thing I noticed right away is that you are not implementing using blocks. That adds a randomness factor that might be multiplied by the number of requests, so I suggest you fix that:

var req = WebRequest.Create(url);
using (WebResponse resp = req.GetResponse())
{
    using (Stream s = resp.GetResponseStream())
    {
        using (var sr = new StreamReader(s, Encoding.ASCII))
        {
            return sr.ReadToEnd();
        }
    }
}

Next, FYI, Fiddler acts as a proxy. If your default proxy was set up to use a script to set up the proxy configuration, then I wonder whether having Fiddler running might not remove the time necessary to do the script setup. That might happen only once, rather than on each request.

Answer 2

Looks like I've now been able to get the throughput right up (to double that I was getting with Fiddler open actually) by setting the max connections in the App.config:

<system.net>
  <connectionManagement>
    <add address="*" maxconnection="30" />
  </connectionManagement>
</system.net>

Very happy with the result but am still a little mystified as to why having Fiddler open changed the results so dramatically.

Answer 3

I had a problem similar to yours and wanted to share my resolution.

In short, I had a console program that was making HTTP requests and would, after 15 minutes or so, timeout. However, if I used Fiddler then I never experienced timeouts, even after having it run for days straight.

I tried setting the maxconnections property in App.config, but that didn't seem to help at all. I then went in and each and every reference to HttpWebRequest, HttpWebResponse, and the stream objects used to read/write data to these objects within using blocks.

That seems to have done the trick. I've been running for almost 24 hours now without a timeout and without Fiddler running.

Answer 4

The way you query causes to create a new session for each call, which is overhead, it could be that fiddler adds session to your queries....

try

private static CookieContainer _cookieContainer = new CookieContainer();

_httpWebRequest.CookieContainer = _cookieContainer; //with recycling the cookiecontainer