tags:

views:

195

answers:

3

Say I have an IO-bound task. I'm using WithDegreeOfParallelism = 10 and WithExecution = ForceParallelism mode, but still the query only uses two threads. Why?

I understand PLINQ will usually choose a degree of parallelism equal to my core count, but why does it ignore my specific request for higher parallelism?

static void Main(string[] args)
{
    TestParallel(0.UpTo(8));
}

private static void TestParallel(IEnumerable<int> input)
{
    var timer = new Stopwatch();
    timer.Start();
    var size = input.Count();

    if (input.AsParallel().
        WithDegreeOfParallelism(10).
        WithExecutionMode(ParallelExecutionMode.ForceParallelism).
        Where(IsOdd).Count() != size / 2)
        throw new Exception("Failed to count the odds");

    timer.Stop();
    Console.WriteLine("Tested " + size + " numbers in " + timer.Elapsed.TotalSeconds + " seconds");
}

private static bool IsOdd(int n)
{
    Thread.Sleep(1000);
    return n%2 == 1;
}
+5  A: 

PLINQ tries to find the optimal number of threads to perform what you want it to do as quickly as possible, if you only have 2 cores on your cpu, that number is most likely 2. If you had a quad core, you would be more likely to see 4 threads appear, but creating 4 threads on a dual core machine wouldn't really improve performance because only 2 threads could be active at the same time.

Also, with IO-based operations, it is likely that any extra threads would simply block on the first IO operation performed.

Rory
Doesn't really answer my question - why does it choose to use two threads even though I specifically request a degree of parallelism = 10 ? (Updated question)
ripper234
@ripper234: From the MSDN documentation: "Degree of parallelism is the **maximum** number of concurrently executing tasks that will be used to process the query". `WithDegreeOfParallelism` is just a hint that PLINQ should use *no more* than *n* threads. http://msdn.microsoft.com/en-us/library/dd383719%28VS.100%29.aspx
LukeH
So ... there's no way to effectively use PLINQ for IO-bound tasks?
ripper234
@ripper234: The "Introduction to PLINQ" article on MSDN specifically suggests bumping up the degree of parallelism for I/O bound tasks: "In cases where a query is performing a significant amount of non-compute-bound work such as File I/O, it might be beneficial to specify a degree of parallelism greater than the number of cores on the machine". But I suppose that PLINQ itself makes the final decision and if it decides (rightly or wrongly) that increasing the parallelism won't help performance then it won't do it! http://msdn.microsoft.com/en-us/library/dd997425%28VS.100%29.aspx
LukeH
+2  A: 

10 is maximum

Sets the degree of parallelism to use in a query. Degree of parallelism is the maximum number of concurrently executing tasks that will be used to process the query.

From here:

MSDN

Kugel
A: 

It appears PLINQ tunes the number of threads. When I wrapped the above code in a while(true) loop, the first two iteration took two seconds to run, but the third and above took only one second. PLINQ understood the cores are idle and upped the number of threads. Impressive!

ripper234
Note that for this to happen, you really have to specify WithDegreeOfParallelism, otherwise PLINQ would limit itself to the number of cores on your machine.
ripper234