What is the best way to download files via HTTP using .NET?

Question

In one of my application I'm using the WebClient class to download files from a web server. Depending on the web server sometimes the application download millions of documents. It seems to be when there are lot of documents, performance vise the WebClient doesn't scale up well.

Also it seems to be the WebClient doesn't immediately close the connection it opened for the WebServer even after it successfully download the particular document.

I would like to know what other alternatives I have.

Update: Also I noticed that for each and every download WebClient performs the authentication hand shake. I was expecting to see this hand shake once since my application only communicate with a single web server. Shouldn't the subsequent calls of the WebClient reuse the authentication session?

Update: My application also calls some web service methods and for these web service calls it seems to authentication session is reused. Also I'm using WCF to communicate with the web service.

Answer 1

WebClient is probably the best option. It doesn't close the connection straight away for a reason: so it can use the same connection again, without having to open a new one. If you find that it's not reusing the connection as expected, that's usually because you're not Close()ing the response from the previous request:

var request = WebRequest.Create("...");
// populate parameters

var response = request.GetResponse();
// process response

response.Close(); // <-- make sure you don't forget this!

Answer 2

I think you can still use "WebClient". However, you are better off using the "using" block as a good practice. This will make sure that the object is closed and is disposed off:-

using(WebClient client = new WebClient()) {
// Use client
} 

Answer 3

I bet you are running into the default limit of 2 connections per server. Try running this code at the beginning of your program:

var cme = new System.Net.Configuration.ConnectionManagementElement();
cme.MaxConnection = 100;
System.Net.ServicePointManager.DefaultConnectionLimit = 100;

Answer 4

I have noticed the same behavior with the session in another project I was working on. To solve this "problem" I did use a static CookieContainer (since the session of the client is recognized by a value saved in a cookie).

public static class SomeStatics
{ 
    private static CookieContainer _cookieContainer;
    public static CookieContainer CookieContainer
    {
         get
         {
             if (_cookieContainer == null)
             {
                 _cookieContainer = new CookieContainer();
             }
            return _cookieContainer;
         }
    }
}

public class CookieAwareWebClient : WebClient
{ 
    protected override WebRequest GetWebRequest(Uri address)
    {
        WebRequest request = base.GetWebRequest(address);
        if (request is HttpWebRequest)
        {
            (request as HttpWebRequest).CookieContainer = SomeStatics.CookieContainer;
            (request as HttpWebRequest).KeepAlive = false;
        }
        return request;
    }
}

//now the code that will download the file
using(WebClient client = new CookieAwareWebClient())
{
    client.DownloadFile("http://address.com/somefile.pdf", @"c:\\temp\savedfile.pdf");
}

The code is just an example and inspired on http://stackoverflow.com/questions/1777221/c-using-cookiecontainer-with-webclient-class and http://stackoverflow.com/questions/896207/c-get-rid-of-connection-header-in-webclient.

The above code will close your connection immediately after the file is download and it will reuse the authentication.