What's the easiest way of making several web requests in a row ?

Question

I have a given list of URL's, and i make an HTTP web request object, and try to connect with it, i have an 'array' of url's, and i try to connect with each one. the objective is seeing which ones are out.

It already works, but one request only starts as soon as the last one ends, so it's quite a slow working, about two requests per second.

I was wondering if i should make about 5 threads working alongside in the background, that would make it 5 times faster, which is the desired speed (whithout overloading the shared internet band). But i have two problems:

1 - i don't even know IF it is the best solution for my problem. 2 - i've tried some times, but i'm new to .NET framework, and have never used multi-thread. so i don't know how i would do it easily.

I have a function start(), and it has a For that goes through all the url's checking existence.

data: VS 08, .NET 3.5, C#.

--[edit]--

Can anyone tell me (with code sample if possible) how to use five (not as many as possible) threads in backgroundworker ? what about starting right after the last processing ends ?

Answer 1

Research the BackgroundWorker Object. This will allow you to spawn multiple thread workers to instantiate asynchronous web requests. Then just use the ReportProgress method to report back on the status of each request.

http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx http://www.dotneat.net/2009/02/10/BackgroundworkerExample.aspx http://dotnetperls.com/backgroundworker

Answer 2

This is an ideal case for the BackgroundWorker class in .NET. It makes use of the thread pool to execute potentially long-running operations in the background so the caller does not have to deal with individual thread creation code.

Answer 3

I prefer to create a worker class that does the HttpWebRequest and start it in its own thread for each connection. Have your worker class use a callback method to signal that its finished. I use a queue of pending threads and a dictionary of active threads. Threads that prematurely finish due to restartable things like connection failures and timeouts can be put back into the queue. The thread's ManagedThreadId is handy for keeping track of threads.

You probably also want to increase your app's maximum number of connections by adding this to your app.config:

  <system.net>
    <connectionManagement>
      <remove address="*"/>
      <add address="*" maxconnection="10" />
    </connectionManagement>
  </system.net>

I chose 10 as an example - you'll have to experiment to see the effect on throughput, CPU utilization, and memory usage.

Answer 4

Rather than explicitly starting new threads yourself, use the HttpWebRequest.BeginGetResponse() method to execute each request asynchronously, and specifying a callback method:

http://www.developerfusion.com/code/4654/asynchronous-httpwebrequest/

Remember to call response.Close() in the callback method so that the maximum number of concurrent connections is not exceeded. This is preferable to increasing the maxconnection value the app.config, which is against RFC guidelines (max connections = 2).

As a rough guide, I can execute around 8 requests per second over a 5 second period using this method.