Simplest way to process a list of items in a multi-threaded manner

Question

I've got a piece of code that opens a data reader and for each record (which contains a url) downloads & processes that page.

What's the simplest way to make it multi-threaded so that, let's say, there are 10 slots which can be used to download and process pages in simultaneousy, and as slots become available next rows are being read etc.

I can't use WebClient.DownloadDataAsync

Here's what i have tried to do, but it hasn't worked (i.e. the "worker" is never ran):

        using (IDataReader dr = q.ExecuteReader())
        {
            ThreadPool.SetMaxThreads(10, 10);
            int workerThreads = 0;
            int completionPortThreads = 0;
            while (dr.Read())
            {
                do
                {
                    ThreadPool.GetAvailableThreads(out workerThreads, out completionPortThreads);

                    if (workerThreads == 0)
                    {
                        Thread.Sleep(100);
                    }
                } while (workerThreads == 0);

                Database.Log l = new Database.Log();
                l.Load(dr);

                ThreadPool.QueueUserWorkItem(delegate(object threadContext)
                {
                    Database.Log log = threadContext as Database.Log;
                    Scraper scraper = new Scraper();
                    dc.Product p = scraper.GetProduct(log, log.Url, true);
                    ManualResetEvent done = new ManualResetEvent(false);
                    done.Set();
                }, l);
            }
        }

Answer 1

You do not normally need to play with the Max threads (I believe it defaults to something like 25 per proc for worker, 1000 for IO). You might consider setting the Min threads to ensure you have a nice number always available.

You don't need to call GetAvailableThreads either. You can just start calling QueueUserWorkItem and let it do all the work. Can you repro your problem by simply calling QueueUserWorkItem?

You could also look into the Parallel Task Library, which has helper methods to make this kind of stuff more manageable and easier.