C# Multi-Threading - What method to use?

views:

487

answers:

+2 Q:

C# Multi-Threading - What method to use?

My goal is to write a program that handles an arbitrary number of tasks based on given user input.

Let's say the # of tasks are 1000 in this case.

Now, I'd like to be able to have a dynamic number of threads that are spawned and start handling the tasks one by one.

I would assume I need to use a "synchronous" method, as opposed to a "asynchronous" one, so that in case one tasks has a problem, I wouldn't want it to slow down the completion of the rest.

What method would I use to accomplish the above? Semaphores? ThreadPools? And how do I make sure that a thread does not try to start a task that is already being handled by another thread? Would a "lock" handle this?

Code examples and/or links to sites that will point me in the right direction will be appreciated.

edit: The problem with the MSDN Fibonacci example is that the waitall method can only handle up to 64 waits. I need more than that due to the 1000 tasks. How to fix that situation without creating deadlocks?

+2 A:

You'll probably want to use the ThreadPool to manage this.

I recommend reading up on MSDN on How to use the ThreadPool in C#. It covers many aspects of this, including firing tasks, and simple synchronization.

Using Threading in C# is the main section, and will cover other options.

If you happen to be using VS 2010 beta, and targetting .NET 4, the Task Parallel Library is a very good option for this - it simplifies some of these patterns.

Reed Copsey 2009-09-20 18:56:37

The problem with that example is that the waitall method can only handle up to 64 waits. I need more than that due to the 1000 tasks.

Sev 2009-09-20 18:57:34

You can just use WaitOne in a loop, and wait on each of element in a List or array.

Reed Copsey 2009-09-20 18:59:06

The WaitOne would not cause issues with deadlocks?

Sev 2009-09-20 19:01:35

Nope. Works the same as WaitAll.

Reed Copsey 2009-09-20 19:58:55

+7 A:

Are these tasks independent? If so, you basically want a producer/consumer queue or a custom threadpool, which are effectively different views on the same thing. You need to be able to place tasks in a queue, and have multiple threads be able to read from that queue.

I have a custom threadpool in MiscUtil or there's a simple (nongeneric due to age) producer/consumer queue in my threading tutorial (about half way down this page).

If these tasks are reasonably long-running, I wouldn't use the system threadpool for this - it will spawn more threads than you probably want. If you're using .NET 4.0 beta 1 you could use Parallel Extensions though.

I'm not quite sure about your comment on WaitAll... are you trying to work out when everything's finished? In the producer/consumer queue case, that would probably involve having some sort of "stop" entry in the queue (e.g. null references which the consuming threads would understand to mean "quit") and then add a "WaitUntilEmpty" method (which should be fairly easy to implement). Note that you wouldn't need to wait until the last items had been processed, as they'd all be stop signals... by the time the queue has emptied, all the real work items will definitely have been processed anyway.

Jon Skeet 2009-09-20 18:59:10

They are independent.

Sev 2009-09-20 19:01:04

I was just looking at this example: http://msdn.microsoft.com/en-us/library/3dasc8as%28VS.80%29.aspx -- which uses a WaitAll, and that method only handles up to 64 -- which is problematic in my case. Using WaitOne I read causes deadlock issues. But you're suggesting I don't need to necessarily wait until they're all done (which is a possibility in my case)

Sev 2009-09-20 19:04:14

@Sev: I'm saying that waiting for everything to be finish can usually be a lot easier than waiting for each individual task to finish :)

Jon Skeet 2009-09-20 19:16:06

+1 for producer/consumer queue. That sounds exactly like what the OP wants.

Daniel Pryden 2009-09-20 19:22:19

Thanks Jon, I think producer/consumer queue will do the trick. And thank you for clarifying.

Sev 2009-09-20 19:34:35

@Jon, does this method just use 1 thread to handle all the jobs? My goal was to have multiple threads handling the tasks.

Sev 2009-09-20 19:39:37

@Jon, speaking of your example here: http://pobox.com/~skeet/csharp/threads/deadlocks.shtml

Sev 2009-09-20 19:40:15

@Sev: yes, that example only uses one consumer thread, but just start up more threads doing the same thing. You'll need to add as many "stop" tokens as you've got consumer threads.

Jon Skeet 2009-09-20 19:51:20

@Jon: Which one is the "stop" token in that example?

Sev 2009-09-20 20:18:42

@Sev: My example doesn't use a stop token, but I'd recommend just using a null reference: the consumer thread would just quit if it saw one. The consumer job would normally be in a `while (true)` loop, rather than one with an explicit counter. Just break the loop if the value retrieved is null.

Jon Skeet 2009-09-20 20:21:35

A simple strategy to avoid a task is get by two or more thread is a syncronized (with a mutext for example) vector.

See this http://msdn.microsoft.com/en-us/library/yy12yx1f.aspx

Matteo Valdina 2009-09-20 19:01:32

Perhaps you can use the BackgroundWorker class. It creates a nice abstraction on top of the thread pool. You can even subclass it if you want to setup many similar jobs.

Martin Liversage 2009-09-20 20:41:32

As has been mentioned, .NET 4 features the excellent Task Parallel Library. But you can use the June 2008 CTP of it in .NET 3.5 just fine. I've been doing this for some hobby projects myself, but if this is a commercial project, you should check out if there are legal issues.

JulianR 2009-09-20 21:01:45

ansaurus

tags:

views:

answers:

C# Multi-Threading - What method to use?

related questions