BackgroundWorker limitation on Windows server 2003

Question

I am currently having an issue with BackgroundWorker running on Windows Server 2003. I have a window application that need to run more than 50 threads.

The code I wrote use BackgroundWorker(BW) as a Thread wrapper to update data onto a window form. The issue is that the code are able to run more than 50 BWs on my XP machine, but stop at 50 when running on Windows 2003 server.

At first I though there are some kind of limit on the number of thread per app can run. Googling the issue shows that is not the case. I wrote the following code to confirm that.

static int count = 0;

static void Main(string[] args)
{
    int max = 55; // default value

    if (args.Length > 0)
            // use command line parameter if provided
     max = Convert.ToInt32(args[0]); 

    List<Thread> threadList = new List<Thread>();
    try
    {
     while (count < max)
     {
      Thread newThread = new Thread(
                        new ParameterizedThreadStart(DummyCall), 1024);
      newThread.Start(count);

      threadList.Add(newThread);
      count++;
     }
    }
    catch (Exception ex)
    {
     Console.WriteLine(ex.ToString());
    }

    Console.ReadLine();
}

static void DummyCall(object obj)
{
    Console.WriteLine(obj.ToString());
    Thread.Sleep(1000000000);
}

The result show as expected. I can see a list of number from 0 to 54 on both my XP machine and the 2003 server.

However, when I try using the BW instead, my XP machine run to 54 and the 2003 server run to 49 (50 BWs). Here is the code.

static int count = 0;

static void Main(string[] args)
{
    int max = 55; // default value

    if (args.Length > 0)
            // use command line parameter if provided
     max = Convert.ToInt32(args[0]); 

    List<BackgroundWorker> list = new List<BackgroundWorker>();

    try
    {
     while (count < max)
     {
      BackgroundWorker worker = new BackgroundWorker();
      worker.DoWork += new DoWorkEventHandler(worker_DoWork);
      worker.RunWorkerAsync(count);

      list.Add(worker);

      count++;
     }
    }
    catch (Exception ex)
    {
     Console.WriteLine(ex.ToString());
    }

    Console.ReadLine();
}

static void worker_DoWork(object sender, DoWorkEventArgs e)
{
    Console.WriteLine(e.Argument.ToString());
    Thread.Sleep(1000000000);
}

So, the question is that why there is a limit on number of BW instances can run on the 2003 server but not XP? Is there anyway I can increase the number of BW instances on 2003 server? If yes, how can I do that?

Answer 1

BackgroundWorker is an inappropriate choice for the kind of work you're doing - it's intended as a means to spin off a (usually single) worker thread while keeping your UI responsive. the worker threads are provided with easy means to interact with your UI - I doubt you have 50 threads all touching your UI.

Consider using threads directly instead. The ThreadPool makes this easy. From MSDN:

using System;
using System.Threading;
public class Example {
    public static void Main() {
        // Queue the task.
        ThreadPool.QueueUserWorkItem(new WaitCallback(ThreadProc));

        Console.WriteLine("Main thread does some work, then sleeps.");
        Thread.Sleep(1000);

        Console.WriteLine("Main thread exits.");
    }

    // This thread procedure performs the task.
    static void ThreadProc(Object stateInfo) {
        // No state object was passed to QueueUserWorkItem, so 
        // stateInfo is null.
        Console.WriteLine("Hello from the thread pool.");
    }
}

You're seeing a 50-thread limit because some versions of the CLR have a default limit of 25 ThreadPool threads per core. Thus, on a dual-core CPU, 50 threads. You can bump this up with ThreadPool.SetMaxThreads. I believe that newer versions of the CLR set the default much higher. Note also that the ThreadPool throttles thread creation, to something like one new thread per 500ms.

Are your two systems running exactly the same version of the framework (including SP)? Do they have the same number of cores?

Note that threads have a considerable overhead, and it's often not worthwhile to create significantly more threads than you have cores. If your bottleneck is off-CPU (say, you're talking to multiple remote systems), then having dozens of threads might be worthwhile. If you're doing CPU-bound calculations, then it's not. If you're disk-IO-bound, be careful that you're not thrashing your disk with that number of parallel operations.

Answer 2

As Michael Petrotta said, the BackgroundWorker is really just for a single worker thread to keep your UI responsive.

The ThreadPool is useful if you have a lot of short-lived actions, but not really intended to run many long-lived actions. The default max number of threads depends on the CLR version (250 per CPU starting with 2.0 SP1, 25 per CPU before). You can increase this number with ThreadPool.SetMaxThreads, but I wouldn't recommend this.

If you have 50+ long-lived, CPU-bound actions that need to be executed in parallel, I think the best option is to create a Thread for each and use an AsyncOperation (SynchronizationContext) to update your UI.