Lightweight Thread Pool Libraries in .NET

Question

I'm looking for a thread pool library in .NET

• I should able to push about 100.000 tasks, therefore should provide support for "blocking" . (obviously I can't push so many tasks at once, so should support some blocking while adding new tasks, and should block the thread until a new slot is available)
• Not overly complicated
• Not so expensive (memory - CPU wise)
• Free / Open Source (Commercial Friendly License)

I've already got my own solution which supports all of these, but I'm not quite sure if it's the best implementation or not. So I want to look into other libraries.

This will be mainly used for HTTP Requests, so less CPU more response wait. That means 100 concurrent thread is acceptable.

Answer 1

Why not just use the .NET ThreadPool?

Answer 2

If you have that many tasks, consider implementing a service bus or 3rd-party middleware platform like BizTalk. If that's too much, then plain Message Queuing.

Also, if you have SQL Server and want the queue to be more easily backed up and managed, consider the SQL Server Service Broker.

Answer 3

The best i've seen recently is Jeffery Richter's PowerThreading library. You might also want to check out this Channel 9 video

The nice thing about the PowerThreading library is that it makes efficient use of the the ThreadPool by allowing a single thread to be shared across many requests.

Answer 4

I found an interesting take on the ThreadPool by this fellow on CodeProject. It is an instantiated ThreadPool (thus, not static and shred across threads) and it has several features to customize performance.

Answer 5

Retlang rocks! Or Rhino Queues from Ayende.

Answer 6

Check out Microsoft's CCR. It has a very efficient thread-pool and has primitives that make dealing with many oustanding asynchronous I/O requests straightforward.

Under its model, providng your I/O is truly asynchronous, the number of threads you'd need would be exactly the number of cores on your box. Anything more is a waste.

Answer 7

I had the same necessity as you but my question was phrased in a different way to achieve very similar answers: http://stackoverflow.com/questions/484511/is-there-any-way-for-executing-a-method-multiple-times-but-managing-connections

Answer 8

.Net 4.0 is adding the Task Parallel Library which will have direct support for this. If you're not aware of it, it is available both as a standalone CTP based on .Net 3.5 and as part of the Visual Studio 2010 CTP that was made available last fall. Here's a link to the team blog which has more information and a pointer to the download.

Answer 9

Thanks for the all replies, after going through other libraries I noticed them all of are quite over-complicated for what I was looking.

I stick with my own implementation which is basically a blocking queue implementation around Threading.Thread() and it works just fine.