Multi:Threading - Is this the right approach?

Question

Experts -

I need some advice in the following scenario.

I have a configuration file with a list of tasks. Each task can have zero, one or more dependencies. I wanted to execute these tasks in parallel [right now they are being executed sequentially]

The idea is to have a main program to read the configuration file and load all the tasks. Read individual tasks and give it to an executor [callable] that will perform the task and return results in a Future. When the task is submitted to the executor (thread) it will monitor for its dependencies to finish first and perform its own task.

Is this the right approach? Are there any other better approaches using java 1.5 features?

Answer 1

What you have described is the Java5/6 approach.

Just make sure that your tasks/Callables are threadsafe / don't share state.

Answer 2

Sounds fine, but beware of Thread starvation deadlock. Basically, don't use a bounded thread pool.

Here is a example that illustrates this problem.
http://www.javaconcurrencyinpractice.com/listings/ThreadDeadlock.java

Also, if you have e.g. a pooled DB connection, you might run into problem, too. 10 threads can block, holding all the pooled connection, waiting for the 11th thread that can't acquire the 11th pooled connection because there isn't anymore available..

Answer 3

Consider using ValueFuture (from guava) to communicate between tasks. Add a listener to each ValueFuture which submits a new task to the ExecutorService only when all of its inputs are available. That way you will never have a thread block part-way through a task waiting for another task to finish.

Answer 4

Database activity, archiving and rule processing are perfect uses for Ant (or even Maven)

Don't reinvent this. Just use Ant.

Write Ant Tasks for each task. Define the dependencies in XML, using Ant's dependency rules.

Turn Ant loose to run your tasks based on the dependencies. You can start out with "executable" Ant Tasks (http://ant.apache.org/manual/tasksoverview.html#exec) rather than defining your own subclass of Task.

Ant understands parallel tasks. You don't need to do anything but declare the tasks as parallel. http://ant.apache.org/manual/CoreTasks/parallel.html

Don't invent any new software for this. Use Ant.