I am trying to build a pipeline of jobs to be submitted to lsf job queues but the documentation left me somewhat guessing as to how this can be implemented.
Let's say I have 3 jobs A, B and C, where B depends on A being finished successfully and C depends on B. I know about the -w option for bsub, with which I can define a dependency such as
bsub -J "B" -w "A"
to submit job B and make it dependant on job A beeing "done". So far, so good, but what happens if Job A fails? What I can see in the queues is that job B is then kept as "PENDING" and
bsub -l B
will now correctly tell me that
Dependency condition invalid or never satisfied;
as the job's dependencies cannot be fulfilled anymore. Unfortunately, the job still stays in the queue and it doesn't look like it is ever cleaned up automatically (which I find a bit puzzling given that lsf already figured out that it will never be run anyway).
Simliarly, I know that I can run a job with a "post_err" dependency that could potentially clean up the above job pipeline by issuing a bkill for all jobs in it. But it seems that, if everything works fine, that job will now be kept in the queue forever with a dpendency that cannot be satisfied because the job it monitors did not exist with an error.
To cut a long story short: how do others handle pipelines of jobs on lsf?