C++ master/worker

Question

I am looking for a cross-platform C++ master/worker library or work queue library. The general idea is that my application would create some sort of Task or Work objects, pass them to the work master or work queue, which would in turn execute the work in separate threads or processes. To provide a bit of context, the application is a CD ripper, and the the tasks that I want to parallelize are things like "rip track", "encode WAV to Mp3", etc.

My basic requirements are:

• Must support a configurable number of concurrent tasks.
• Must support dependencies between tasks, such that tasks are not executed until all tasks that they depend on have completed.
• Must allow for cancellation of tasks (or at least not prevent me from coding cancellation into my own tasks).
• Must allow for reporting of status and progress information back to the main application thread.
• Must work on Windows, Mac OS X, and Linux
• Must be open source.

It would be especially nice if this library also:

• Integrated with Qt's signal/slot mechanism.
• Supported the use of threads or processes for executing tasks.

By way of analogy, I'm looking for something similar to Java's ExecutorService or some other similar thread pooling library, but in cross-platform C++. Does anyone know of such a beast?

Thanks!

Answer 1

This is probably a huge overkill for what you need but still worth mentioning -
BOINC is a distributed framework for such tasks. There's a main server that gives out tasks to perform and a cloud of workers that do its bidding. It is the framework behind projects like SETI@Home and many others.

Answer 2

Sounds like you require some kind of "Time Sharing System".

There are some good open source ones out there, but I don't know
if they have built-in QT slot support.

Answer 3

I haven't used it in long enough that I'm not positive whether it exactly meets your needs, but check out the Adaptive Communications Environment (ACE). This library allows you to construct "active objects" which have work queues and execute their main body in their own threads, as well as thread pools that can be shared amoung objects. Then you can pass queue work objects on to active objects for them to process. Objects can be chained in various ways. The library is fairly heavy and has a lot to it to learn, but there have been a couple of books written about it and theres a fair amount of tutorial information available online as well. It should be able to do everything you want plus more, my only concern is whether it possesses the interfaces you are looking for 'out of the box' or if you'd need to build on top of it to get exactly what you are looking for.

Answer 4

See this post for creating threads using the boost library in C++:

http://stackoverflow.com/questions/266168/simple-example-of-threading-in-c

(it is a c++ thread even though the title says c)

basically, create your own "master" object that takes a "runnable" object and starts it running in a new thread.

Then you can create new classes that implement "runnable" and throw them over to your master runner any old time you want.

Answer 5

I think this calls for intel's Threading Building Blocks, which pretty much does what you want.

Answer 6

Check out Intels' Thread Building Blocks library.