ThreadPool - WaitAll 64 Handle Limit

Question

I am trying to bypass the the wait64 handle limit that .net 3.5 imposes

I have seen this thread : http://stackoverflow.com/questions/2702545/workaround-for-the-waithandle-waitall-64-handle-limit

So I understand the general idea but I am having difficulty because I am not using a delegate but rather

I am basically working of this example : http://msdn.microsoft.com/en-us/library/3dasc8as%28VS.80%29.aspx

This link http://www.switchonthecode.com/tutorials/csharp-tutorial-using-the-threadpool is similar but again the int variable keeping track of the tasks is a member variable.

Where in the above example would I pass the threadCount integer? Do I pass it in the callback method as an object? I think I am having trouble with the callback method and passing by reference.

Thanks Stephen,

That link is not entirely clear to me.

Let me post my code to help myself clarify:

for (int flows = 0; flows < NumFlows; flows++)
{
ResetEvents[flows] = new ManualResetEvent(false);
ICalculator calculator = new NewtonRaphson(Perturbations);
Calculators[flows] = calculator;
ThreadPool.QueueUserWorkItem(calculator.ThreadPoolCallback, flows);
}
resetEvent.WaitOne();

Where would I pass in my threadCount variable. I assume it needs to be decremented in calculator.ThreadPoolCallback?

Answer 1

An anonymous method might be easiest:

int threadCount = 0;
for (int flows = 0; flows < NumFlows; flows++)
{
    ICalculator calculator = new NewtonRaphson(Perturbations);
    Calculators[flows] = calculator;

    // We're about to queue a new piece of work:
    //    make a note of the fact a new work item is starting
    Interlocked.Increment(ref threadCount);
    ThreadPool.QueueUserWorkItem(
        delegate
        {
            calculator.ThreadPoolCallback(flows);

            // We've finished this piece of work...
            if (Interlocked.Decrement(ref threadCount) == 0)
            {
                // ...and we're the last one.
                // Signal back to the main thread.
                resetEvent.Set();
            }
        }, null);
}
resetEvent.WaitOne();

Answer 2

You should not be using multiple wait handles to wait for the completion of multiple work items in the ThreadPool. Not only is it not scalable you will eventually bump into the 64 handle limit imposed by the WaitHandle.WaitAll method (as you have done already). The correct pattern to use in this situation is a counting wait handle. There is one available in the Reactive Extensions download for .NET 3.5 via the CountdownEvent class.

var finished = new CountdownEvent(1);
for (int flows = 0; flows < NumFlows; flows++) 
{ 
  finished.AddCount();
  ICalculator calculator = new NewtonRaphson(Perturbations); 
  Calculators[flows] = calculator; 
  ThreadPool.QueueUserWorkItem(
    (state) =>
    {
      try 
      { 
        calculator.ThreadPoolCallback(state); 
      }
      finally 
      { 
        finished.Signal(); 
      }
    }, flows);
} 
finished.Signal();
finished.Wait();