How can I run multiple jobs from a large queue at the same time in Perl?

Question

I would like to program bsub job limits into my Perl script which launches lsf jobs under the hood. If I have something like 2000 jobs, I would like to run at max 20 jobs at any given time. I would prefer that the jobs running is max 20. I seen scripts where people launch 20 jobs and wait for it to finish before launching another 20.

Answer 1

If you're using Platform LSF or Lava, you could create a job group for your jobs and set a limit that way. The batch system will take care of the rest. You'll have a couple thousand jobs PENDing, but only 20 at at time (or whatever you've set your limit to) RUNning at any one time. Check the docs for the bgadd, bgmod and bjgroup commands.