Most efficient way for testing links

Question

I'm currently developping an app that is going through all the files on a server and checking every single hrefs to check wether they are valid or not. Using a WebClient or a HttpWebRequest/HttpWebResponse is kinda overkilling the process because it downloads the whole page each time, which is useless, I only need to check if the link do not return 404.

What would be the most efficient way? Socket seems to be a good way of doing it, however I'm not quite sure how this works.

Thanks for sharing your expertise!

Answer 1

The most efficient would be to send the HEAD verb to preserve bandwidth.

var request = WebRequest.Create("http://google.com/");
request.Method = "HEAD";
using (var response = (HttpWebResponse)request.GetResponse())
{
    if (response.StatusCode == HttpStatusCode.OK)
    {
        // 200 OK
    }
}

Answer 2

Set HttpWebRequest.Method to "Head". This will only receive the headers.