How to get the maximum outbound requests when parellellizing asynchronous calls?

Question

Analysing the code below in action through Fiddler, I realized that using Parallel Extensions I can get at maximum 2 outbound requests:

new string[] 
    {
        "http://stackoverflow.com",
        "http://superuser.com",
        "http://serverfault.com",
        "http://stackexchange.com"
    }
    .AsParallel()
    .Select(a => HttpWebRequest.Create(a).GetResponse())
    .ToArray()
    ;

What method should I use to maximize the number of outbound requests?

Answer 1

ServicePointManager.DefaultConnectionLimit

http://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.defaultconnectionlimit.aspx

In your case, set it to 4.

Answer 2

By default the PFX creates that number of threads as your CPU number of cores. That why you have only two requests. Use Task class from PFX and run them all thru Task.WaitAll. (I hope my guess is correct.)

EDIT: Example

var tasks = servers.Select(Task.Create(() => GetResponseCallHere(...))).ToArray();
Task.WaitAll(tasks);

Answer 3

Note that in the question I used the sychronous version of the BeginGetResponse method. What I have found so far is that the only way to maximize the outbound request is to use the async version of the method. But this approach yields new problems, namely:

1. You must split your caller logic in two methods
2. It's difficult to keep track whether all outbound calls were completed.
3. It's difficult to handle exceptions.

4. If the number of tasks to run surpasses the thread pool size, then god knows what happens

   new string[] 
   {
       "http://stackoverflow.com",
       "http://superuser.com",
       "http://serverfault.com",
       "http://stackexchange.com"
   }
   .Select(a => HttpWebRequest.Create(a).BeginGetResponse(callback, null))
   .ToArray();
   

Answer 4

This code runs all 6 HTTP requests in parallel on my machine according to Wireshark:

var urls = new string[] 
{
    "http://stackoverflow.com", 
    "http://superuser.com", 
    "http://serverfault.com",
    "http://stackexchange.com",
    "http://www.howtogeek.com",
    "http://meta.stackoverflow.com" 
};

var reqs = urls.Select<string, WebRequest>(HttpWebRequest.Create).ToArray();
var iars = reqs.Select(req => req.BeginGetResponse(null, null)).ToArray();
var rsps = reqs.Select((req, i) => req.EndGetResponse(iars[i])).ToArray();

Basically it creates a WebRequest for each URL, calls BeginGetResponse on each and then calls EndGetResponse for each with the IAsyncResult.

The documentation states that BeginGetResponse uses The Managed Thread Pool to make the HTTP request.