Using multithreading for loop

Question

Hello,

I'm new to threading and want to do something similar to this question:

http://stackoverflow.com/questions/100291/speed-up-loop-using-multithreading-in-c-question

However, I'm not sure if that solution is the best one for me as I want them to keep running and never finish. (I'm also using .net 3.5 rather than 2.0 as for that question.)

I want to do something like this:

foreach (Agent agent in AgentList)
{
    // I want to start a new thread for each of these
    agent.DoProcessLoop();
}

---

public void DoProcessLoop()
{
    while (true)
    {
        // do the processing

        // this is things like check folder for new files, update database
        // if new files found
    }
}

Would a ThreadPool be the best solution or is there something that suits this better?

Update: Thanks for all the great answers! I thought I'd explain the use case in more detail. A number of agents can upload files to a folder. Each agent has their own folder which they can upload assets to (csv files, images, pdfs). Our service (it's meant to be a windows service running on the server they upload their assets to, rest assured I'll be coming back with questions about windows services sometime soon :)) will keep checking every agent's folder if any new assets are there, and if there are, the database will be updated and for some of them static html pages created. As it could take a while for them to upload everything and we want them to be able to see their uploaded changes pretty much straight away, we thought a thread per agent would be a good idea as no agent then needs to wait for someone else to finish (and we have multiple processors so wanted to use their full capacity). Hope this explains it!

Thanks,

Annelie

Answer 1

One issue with ThreadPool would be that if the pool happens to be smaller than the number of Agents you would like to have, the ones you try to start later may never execute. Some tasks may never begin to execute, and you could starve everything else in your app domain that uses the thread pool as well. You're probably better off not going down that route.

Answer 2

Until you upgrade to .NET 4, the ThreadPool might be your best option. You may also want to use a Semaphore and a AutoResetEvent to control the number of concurrent threads. If you're talking about long-running work then the overhead of starting up and managing your own threads is low and the solution is more elegant. That will allow you to use a WorkerThread.Join() so you can make sure all worker threads are complete before you resume execution.

Answer 3

A thread pool is useful when you expect threads to be coming into and out of existence fairly regularly, not for a predefined set number of threads.

Answer 4

You definitely don't want to use the ThreadPool for this purpose. ThreadPool threads are not meant to be used for long-running tasks ("infinite" counts as long-running), since that would obviously tie up resources meant to be shared.

For your application, it would probably be better to create one thread (not from the ThreadPool) and in that thread execute your while loop, inside of which you iterate through your Agents collection and perform the processing for each one. In the while loop you should also use a Thread.Sleep call so you don't max out the processor (there are better ways of executing code periodically, but Thread.Sleep will work for your purposes).

Finally, you need to include some way for the while loop to exit when your program terminates.

Update: Finally finally, multi-threading does not automatically speed up slow-running code. Nine women can't make a baby in one month.

Answer 5

I believe that the Parallels Extensions make this possible:

Parallel.Foreach

http://msdn.microsoft.com/en-us/library/system.threading.tasks.parallel.foreach.aspx http://blogs.msdn.com/pfxteam/

Answer 6

Given the specific usage your describe (watching for files), I'd suggest you use a FileSystemWatcher to determine when there are new files and then fire off a thread with the threadpool to process the files until there are no more to process -- at which point the thread exits.

This should reduce i/o (since you're not constantly polling the disk), reduce CPU usage (since the constant looping of multiple threads polling the disk would use cycles), and reduce the number of threads you have running at any one time (assuming there aren't constant modifications being made to the file system).

You might want to open and read the files only on the main thread and pass the data to the worker threads (if possible), to limit i/o to a single thread.

Answer 7

Hmm.. as Ragoczy points out, its better to use FileSystemWatcher to monitor the files. However, since you have additional operations, you may think in terms of multithreading.

But beware, no matter how many processers you have, there is a limit to it's capacity. You may not want to create as many threads as the number of concurrent users, for the simple reason that your number of agents can increase.