Is this a right case for using the Threadpool?

Question

Here's the setup: I'm trying to make a relatively simple Winforms app, a feed reader using the FeedDotNet library. The question I have is about using the threadpool. Since FeedDotNet is making synchronous HttpWebRequests, it is blocking the GUI thread. So the best thing seemed like putting the synchronous call on a ThreadPool thread, and while it is working, invoke the controls that need updating on the form. Some rough code:

private void ThreadProc(object state)
{
 Interlocked.Increment(ref updatesPending);

    // check that main form isn't closed/closing so that we don't get an ObjectDisposedException exception
 if (this.IsDisposed || !this.IsHandleCreated) return;
 if (this.InvokeRequired)
  this.Invoke((MethodInvoker)delegate
  {
   if (!marqueeProgressBar.Visible)
    this.marqueeProgressBar.Visible = true;
  });

 ThreadAction t = state as ThreadAction;
 Feed feed = FeedReader.Read(t.XmlUri);

 Interlocked.Decrement(ref updatesPending);

 if (this.IsDisposed || !this.IsHandleCreated) return;
 if (this.InvokeRequired)
  this.Invoke((MethodInvoker)delegate { ProcessFeedResult(feed, t.Action, t.Node); });

 // finished everything, hide progress bar
 if (updatesPending == 0)
 {
  if (this.IsDisposed || !this.IsHandleCreated) return;
  if (this.InvokeRequired)
   this.Invoke((MethodInvoker)delegate { this.marqueeProgressBar.Visible = false; });
 }
}

this = main form instance

updatesPending = volatile int in the main form

ProcessFeedResult = method that does some operations on the Feed object. Since a threadpool thread can't return a result, is this an acceptable way of processing the result via the main thread?

The main thing I'm worried about is how this scales. I've tried ~250 requests at once. The max number of threads I've seen was around 53 and once all threads were completed, back to 21. I recall in one exceptional instance of me playing around with the code, I had seen it rise as high as 120. This isn't normal, is it? Also, being on Windows XP, I reckon that with such high number of connections, there would be a bottleneck somewhere. Am I right?

What can I do to ensure maximum efficiency of threads/connections?

Having all these questions also made me wonder whether this is the right case for a Threadpool use. MSDN and other sources say it should be used for "short-lived" tasks. Is 1-2 seconds "short-lived" enough, considering I'm on a relatively fast connection? What if the user is on a 56K dial-up and one request could take from 5-12 seconds and ever more. Would the threadpool be an efficient solution then too?

Answer 1

You may want to take a look to the "BackgroundWorker" class.

Answer 2

My understanding of the ThreadPool is that it is designed for this type of situation. I think the definition of short-lived is of this order of time - perhaps even up to minutes. A "long-lived" thread would be one that was alive for the lifetime of the application.

Don't forget Microsoft would have spent some getting the efficiency of the ThreadPool as high as it could. Do you think that you could write something that was more efficient? I know I couldn't.

Answer 3

The ThreadPool, unchecked is probably a bad idea.

Out of the box you get 250 threads in the threadpool per cpu.

Imagine if in a single burst you flatten out someones net connection and get them banned from getting notifications from a site cause they are suspected to be running a DoS attack.

Instead, when downloading stuff from the net you should build in tons of control. The user should be able to decide how many concurrent requests they make (and how many concurrent requests per domain), ideally you also want to offer controls for the amount of bandwidth.

Though this could be orchestrated with the ThreadPool, having dedicated threads or using something like a bunch of instances of the BackgroundWorker class is a better option.

Answer 4

The .NET thread pool is designed specifically for executing short-running tasks for which the overhead of creating a new thread would negate the benefits of creating a new thread. It is not designed for tasks which block for prolonged periods or have a long execution time.

The idea is to for a task to hop onto a thread, run quickly, complete and hop off.

The BackgroundWorker class provides an easy way to execute tasks on a thread pool thread, and provides mechanisms for the task to report progress and handle cancel requests.

In this MSDN article on the BackgroundWorker Component, file downloads are explicitly given as examples of the appropriate use of this class. That should hopefully encourage you to use this class to perform the work you need.

If you're worried about overusing the thread pool, you can be assured the runtime does manage the number of available threads based on demand. Tasks are queued on the thread pool for execution. When a thread becomes available to do work, the task is loaded onto the thread. At regular intervals, a monitoring process checks the state of the thread pool. If there are tasks waiting to be executed, it can create more threads. If there are several idle threads, it can shut down some to release resources.

In a worse-case scenario, where all threads are busy and you have work queued up, the runtime will be adding threads to deal with the extra workload. The application will be running more slowly as it has to wait for more threads to be made available, but it will continue to run.