How can I measure the response and loading time of a webpage?

Question

I need to build a windows forms application to measure the time it takes to fully load a web page, what's the best approach to do that?

The purpose of this small app is to monitor some pages in a website, in a predetermined interval, in order to be able to know beforehand if something is going wrong with the webserver or the database server.

Additional info:

I can't use a commercial app, I need to develop this in order to be able to save the results to a database and create a series of reports based on this info.

The webrequest solution seems to be the approach I'm goint to be using, however, it would be nice to be able to measure the time it takes to fully load the the page (images, css, javascript, etc). Any idea how that could be done?

Answer 1

Fully load as in also loading all external resources such as CSS, JS, images etc? What about delayed AJAX requests, Flash movies - should they also be loaded? Or do you just wish to measure the TTLB (time to last byte) of the main request?

Answer 2

You can use a software like these :

http://www.cyscape.com/products/bhawk/page-load-time.aspx

http://www.trafficflowseo.com/2008/10/website-load-timer-great-to-monitor.html

Google will be helpful to find the one best suited for your needs.

Answer 3

• http://www.httpwatch.com/
• Firebug NET tab

Answer 4

Something like this would probably work fine:

System.Diagnostics.Stopwatch sw = new Stopwatch()
System.Net.HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("http://www.example.com");
// other request details, credentials, proxy settings, etc...

sw.Start();
System.Net.HttpWebResponse res = (HttpWebResponse)req.GetResponse();
sw.Stop();

TimeSpan timeToLoad = sw.Elapsed;

Answer 5

If you're using firefox install the firebug extension found at http://getfirebug.com. From there, choose the net tab, and it'll let you know the load/response time for everything on the page.

Answer 6

If you just want to record how long it takes to get the basic page source, you can wrap a HttpWebRequest around a stopwatch. E.g.

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(address);

System.Diagnostics.Stopwatch timer = new Stopwatch();
timer.Start();

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

timer.Stop();

TimeSpan timeTaken = timer.Elapsed;

However, this will not take into account time to download extra content, such as images.

[edit] As an alternative to this, you may be able to use the WebBrowser control and measure the time between performing a .Navigate() and the DocumentCompleted event from firing. I think this will also include the download and rendering time of extra content. However, I haven't used the WebBrowser control a huge amount and only don't know if you have to clear out a cache if you are repeatedly requesting the same page.

Answer 7

Depending on how the frequency you need to do it, maybe you can try using Selenium (a automated testing tool for web applications), since it users internally a web browser, you will have a pretty close measure. I think it would not be too difficult to use the Selenium API from a .Net application (since you can even use Selenium in unit tests).

Measuring this kind of thing is tricky because web browsers have some particularities when then download all the web pages elements (JS, CSS, images, iframes, etc) - this kind of particularities are explained in this excelent book (http://www.amazon.com/High-Performance-Web-Sites-Essential/dp/0596529309/).

A homemade solution probably would be too much complex to code or would fail to attend some of those particularities (measuring the time spent in downloading the html is not good enough).

Answer 8

One thing you need to take account of is the cache. Make sure you are measuring the time to download from the server and not from the cache. You will need to insure that you have turned off client side caching.

Also be mindful of server side caching. Suppose you download the pace at 9:00AM and it takes 15 seconds, then you download it at 9:05 and it takes 3 seconds, and finally at 10:00 it takes 15 seconds again.

What might be happening is that at 9 the server had to fully render the page since there was nothing in the cache. At 9:05 the page was in the cache, so it did not need to render it again. Finally by 10 the cache had been cleared so the page needed to be rendered by the server again.

I highly recommend that you checkout the YSlow addin for FireFox which will give you a detailed analysis of the times taken to download each of the items on the page.

Answer 9

I wrote once a experimental program which downloads a HTML page and the objects it references (images, iframes, etc).

More complicated than it seems because there is HTTP negotiation so, some Web clients will get the SVG version of an image and some the PNG one, widely different in size. Same thing for <object>.

Answer 10

I'm often confronted with a quite similar problem. However I take a slight different approach: First of all, why should I care about static content at all? I mean of course it's important for the user, if it takes 2 minutes or 2 seconds for an image, but that's not my problem AFTER I fully developed the page. These things are problems while developing and after deployment it's not the static content, but it's the dynamic stuff that normally slows things down (like you said in your last paragraph). The next thing is, why do you trust that so many things stay constant? If someone on your network fires up a p2p program, the routing goes wrong or your ISP has some issues your server-stats will certainly go down. And what does your benchmark say for a user living across the globe or just using a different ISP? All I'm saying is, that you are benchmarking YOUR point of view, but that doesn't say much about the servers performance, does it?

Why not let the site/server itself determine how long it took to load? Here is a small example written in PHP:

function microtime_float()
{
  list($usec, $sec) = explode(" ", microtime());
  return ((float)$usec + (float)$sec);
}

function benchmark($finish)
{
  if($finish == FALSE){  /* benchmark start*/    
    $GLOBALS["time_start"] = microtime_float();
  }else{ /* benchmark end */    
    $time = microtime_float() - $GLOBALS["time_start"]; 
    echo '<div id="performance"><p>'.$time.'</p></div>';
  }
}

It adds at the end of the page the time it took to build (hidden with css). Every couple of minutes I grep this with a regular expression and parse it. If this time goes up I know that there something wrong (including the static content!) and via a RSS-Feed I get informed and I can act.

With firebug we know the "normal" performance of a site loading all content (development phase). With the benchmark we get the current server situation (even on our cell phone). OK. What next? We have to make certain that all/most visitors are getting a good connection. I find this part really difficult and are open to suggestions. However I try to take the log files and ping a couple of IPs to see how long it takes to reach this network. Additionally before I decide for a specific ISP I try to read about the connectivity and user opinions...