Multiple HTTP request in C#

Question

I need to send about 200 HTTP requests in parallel to different servers and get response. I use HttpWebRequest class in C#. But I don't see good time enhancement when requests are handled in parallel. For example if one request needs 3sec to get response, 2 request in parallel - 6sec, 3 requests - 8 secs, 4 requests - 11sec ... It is not good, I hope that there should be best time, about 10 sec for 200 requests. It looks like only 2-3 requests performs in parallel, but timeout starts immediately after WebRequest object creation. I tried set DefaultConnectionLimit and MaxServicePoints values, but id didn't help. As I understand these parameters for number of requests to one site in parallel. I need requests to different sites.

Code example that I use to test:

ArrayList a = new ArrayList(150);

for (i = 50; i < 250; i++ )
{
   a.Add("http://207.242.7." + i.ToString() + "/");
}

for (i = 0; i < a.Count; i++)
{
    Thread t = new Thread(new ParameterizedThreadStart(performRequest));
    t.Start(a[i]);
}


static void performRequest(object ip)
{
      HttpWebRequest req = (HttpWebRequest)WebRequest.Create((stirng)ip);

      HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
}

Сan anyone ever encountered such a problem? Thank you for any suggestions.

Answer 1

Use asynchronous web requests in stead.

Edit: http://msdn.microsoft.com/en-us/library/86wf6409(VS.71).aspx

Answer 2

Instead of starting up your own threads try using the asynchronous methods of HttpWebRequest such as HttpWebRequest.BeginGetResponse and HttpWebRequest.BeginGetRequestStream.

Answer 3

This might help - http://social.msdn.microsoft.com/forums/en-US/netfxnetcom/thread/1f863f20-09f9-49a5-8eee-17a89b591007

Suggests there is a limit on the number of TCP connections allowed, but that you can increase the limit

Answer 4

Hi,
you can try this :

try
        {
            List<Uri> uris = new List<Uri>();
            uris.Add(new Uri("http://www.google.fr"));
            uris.Add(new Uri("http://www.bing.com"));

            Parallel.ForEach(uris, u =>
           {
               WebRequest webR = HttpWebRequest.Create(u);
               HttpWebResponse webResponse = webR.GetResponse() as HttpWebResponse;
           });
        }
        catch (AggregateException exc)
        {
            exc.InnerExceptions.ToList().ForEach(e =>
                {
                    Console.WriteLine(e.Message);
                });
        }