.NET - downloading multiple pages from a website with a single DNS query

Question

I'm using HttpRequest to download several pages from a website (in a loop). Simplifying it looks like this:

HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(
            "http://sub.domain.com/something/" + someString
        );

HttpWebResponse resp = (HttpWebResponse)req.GetResponse();

//do something

I'm not quite sure actually but every request seems to resolve the address again (I don't know how to test if I'm right). I would like to boost it a little and resolve the address once and then reuse it for all requests. I can't work out how to force HttpRequest into using it, though.

I have tried using Dns.GetHostAddresses, converting the result to a string and passing it as the address to HttpWebRequest.Create. Unfortunately, server returns error 404 then. I managed to google that's probably because the "Host" header of the http query doesn't match what the server expects.

Is there a simple way to solve this?

Answer 1

I doubt that the DNS isn't being cached already to be honest, but there is a way to do what you ask.

After creating the request with the IP address, set the Host property on it to the DNS name. This should solve your 404 problem.

Something that might help you speed up your multiple requests is to set the KeepAlive property to true. This will keep the connection open and allow you to make multiple requests without having to re-establish the connection each time.

Answer 2

The 404 is definately because of the site's "host header" - thousands of sites can be hosted at a single IP address, and the web server uses the domain to figure out which one you want.

Your local computer should be caching the results of the DNS query, so even though it will generate a request each time you access the domain, the request won't even leave your computer after the first time, just using the locally cached lookup results.