What is the correct way in C# to download a file from the Internet and write it on the fly?

Question

Edit: This is done in the Compact Framework, I don't have access to WebClient therefore it has to be done with HttpWebRequests.

I am creating a download manager application that will be able to have concurrent downloads (more than one download at once) and the ability to report the percentage completed and resume the downloads.

This means that I am downloading some bytes into a buffer and then writing the buffer to disk. I just wanted to check what recommended algorithm/procedure is for this.

This is what I have thus far for the main download method:

    private void StartDownload()
    {
        HttpWebRequest webReq = null;
        HttpWebResponse webRes = null;
        Stream fileBytes = null;
        FileStream saveStream = null;

        try
        {
            webReq = (HttpWebRequest)HttpWebRequest.Create(_url);
            webReq.Headers.Add(HttpRequestHeader.Cookie, "somedata");
            webRes = (HttpWebResponse)webReq.GetResponse();

            byte[] buffer = new byte[4096];
            long bytesRead = 0;
            long contentLength = webRes.ContentLength;

            if (File.Exists(_filePath))
            {
                bytesRead = new FileInfo(_filePath).Length;
            }

            fileBytes = webRes.GetResponseStream();
            fileBytes.Seek(bytesRead, SeekOrigin.Begin);

            saveStream = new FileStream(_filePath, FileMode.Append, FileAccess.Write);

            while (bytesRead < contentLength)
            {
                int read = fileBytes.Read(buffer, 0, 4096);
                saveStream.Write(buffer, 0, read);
                bytesRead += read;
            }
            //set download status to complete
            //_parent
        }
        catch
        {
            if (Thread.CurrentThread.ThreadState != ThreadState.AbortRequested)
            {
               //Set status to error.
            }
        }
        finally
        {
            saveStream.Close();
            fileBytes.Close();
            webRes.Close();
            saveStream.Dispose();
            fileBytes.Dispose();
            saveStream = null;
            fileBytes = null;
            webRes = null;
            webReq = null;
        }
    }

Should I be downloading a larger buffer? Should I be writing the buffer to file so often (every 4KB?) Should there be some thread sleeping in there to ensure not all the CPU is used? I think reporting the progress change every 4KB is stupid so I was planning to do it every 64KB downloaded.

Looking for some general tips or anything that is wrong with my code so far.

Answer 1

In the full framework, the simplest way to do this is to use the WebClient class's DownloadFile method, like this:

using(var wc = new WebClient()) {
    wc.DownloadFile(url, filePath);
}

EDIT: To report the download progress, call DownloadFileAsync and listen for the DownloadProgressChanged event. You can also cancel the download by calling the CancelAsync method.

Answer 2

First thing, I would get rid of the finally clause and change the code to use "USING" clauses.

Anything that implements IDisposable should be programmed that way to make sure garbage collection occurs correctly and when it is supposed to.

For example:

using (HttpWebRequest webReq = (HttpWebRequest)HttpWebRequest.Create(_url)) {
    /* more code here... */
}

Second, I wouldn't instantiate my variables at the head with null values (ala Pascal style). See above example.

Third, the download should be in it's own thread which sync's with a call back function in the main thread to report status. Put the sync call in the middle of your while loop.

Answer 3

You can get all the tips you need from existing code thats been published and is freely available.

Such as here: http://www.codeproject.com/KB/IP/MyDownloader.aspx

Better yet, take the existing code and modify it for your needs. (That's why the code gets posted there in the first place.)

Answer 4

If you need to track the progress, use WebClient.DownloadFileAsync along with the DownloadProgressChanged and DownloadFileCompleted events

WebClient wc = new WebClient();
wc.DownloadProgressChanged += wc_DownloadProgressChanged;
wc.DownloadFileCompleted += wc_DownloadFileCompleted;
wc.DownloadFileAsync(sourceUri, localPath);

...

private void wc_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
    ...
}

private void wc_DownloadFileCompleted(object sender, AsyncCompletedEventArgs e)
{
    ...
}

Answer 5

From a user experience perspective, you should be able to answer a lot of these questions by looking at an application like Internet Explorer or Firefox. For example;

1. In Internet Explorer, new data is reported every few kilobytes, up to the one megabyte mark. After that, it is reported in 100 kilobyte increments.
2. How often you write to the buffer depends on whether you're allowing recovery when the connection is dropped. If you're like IE and force the user to start from scratch, it doesn't really matter how often you save your buffer as long as you do it eventually. Set your saving based on "acceptable loss".
3. Your application should obviously not take 100% of the CPU, since that isn't good etiquette in the programming world. Have your threads at least sleep long enough not to dominate the CPU.

Your code, generally speaking, is functional, though it could stand a lot of refactoring to make it a little cleaner/easier to read. You might consider using the WebClient class in .NET, too, but if this is a learning exercise, you're doing it the right way.

Good luck! You're on the right track.