Monitoring/Resumption for Complex Task Graph for Batch Processing

Question

I have a set of batch-processing tasks that need to be run in an automated way on a weekly basis. The weekly basis part isn't hard: I'm using cron to kick things off. The tasks are basically anything that can be initiated with a shell script. The tricky part is that my tasks have a non-trivial dependency graph (B depends A, C depends on B, D depends C and B, etc...). I'm interested in having task monitoring so that I receive notification when something goes wrong with a task. Even better, I'd like to be able to easily resume the task graph from the failure point without rerunning the whole thing.

Now, I can imagine writing something myself to make all this happen, but I have to imagine that someone has already gone through the trouble. Are there any existing libraries or frameworks that would make this task easier? Ideally, I'm looking for something to run in a linux/unix environment.

Answer 1

I would suggest a Makefile to solve your problems.

You'll have to have marker files for the tasks to indicate when they were last completed, and work out some dependencies for the 'root' tasks (those without an internal dependencies).

A : 
    <A's tasks>
    touch A

B : A
    <B's tasks>
    touch B

C : B
    <C's tasks>
    touch C

D : B C
    <D's tasks>
    touch D

Now you just need to work out why A has to be re-done and you'll be able to cron doing this make file and just the appropriate bits will be re-done.