Using HttpWebRequest to download html pages returns ContentLength as -1

Question

Hi,

I'm trying to get a bunch of HTML files downloaded from the internet and using HttpWebRequest to do it.

However when I use

WebRequest.Create("http://www.stackoverflow.com/"); WebRequest.GetResponse();

The HttpWebResponse.ContentLength is always -1. I have to show the progress of these pages downloading, in our existing VB6 version we're using the inet control and it reports the progress nicely.

My code is very similar to this:

http://www.geekpedia.com/tutorial179_Creating-a-download-manager-in-Csharp.html

(The 'Download' method that is)

I'm not sure what I'm doing wrong or whether its normal - theres no Content-Length header sent by the web-server. How do we workout the progress in this case?

Thanks,

Answer 1

The code in the Download() method in the linked article relies on the private members in the class, so although

WebRequest.Create("http://www.stackoverflow.com");

might compile, it's actually calling a method on the class WebRequest, then discarding the result, which is where the request is actually stored. By the time the line

WebRequest.GetResponse();

is called, the code has forgotten all about the response from Create(). (C# is case sensitive, so WebRequest and webRequest are not the same thing; and it's quite conventional to give the name webRequest to an instance of the WebRequest class.) So try putting the lines

private HttpWebRequest webRequest;
private HttpWebResponse webResponse;

outside the method at the top of the class as instances of the WebRequest and WebResponse objects.

Answer 2

If the web server doesn't send a content length, there's no way of knowing how much more data there is, so it's basically impossible to know how far through it you are.

Heck, the server may be generating data dynamically and still not know (when you get the first part of the data) how much more data it has to send.

Anything which claims to be showing progress (as a proportion of complete download size) is either lying, or is downloading the data in a different way which does give them the full size. Of course, if you're downloading multiple files you can always say how many files you've finished downloading and how many you've still got to download (although even in this case the latter number may not be known if you're doing a crawler-style app).

Note that one common approach is to show a progress bar which indicates "I'm making progress" but doesn't try to guess how much is left to download.

Answer 3

Here's a sample using WebClient you could try:

class Program
{
    static void Main(string[] args)
    {
        var client = new WebClient();
        client.DownloadProgressChanged += (sender, e) =>
        {
            Console.WriteLine("{0}% completed", e.ProgressPercentage);
        };
        client.DownloadStringCompleted += (sender, e) =>
        {
            // e.Result contains downloaded string
            Console.WriteLine("finished downloading...");
        };
        client.DownloadStringAsync(new Uri("http://www.stackoverflow.com"));
        Console.ReadKey();
    }
}

If you decide to use it in a WinForms application make sure you call InvokeRequired/Invoke pattern from event handlers as they are not executed on the UI thread.