Trouble with parallel make not always starting another job when one finishes

Question

I'm working on a system with four logical CPS (two dual-core CPUs if it matters). I'm using make to parallelize twelve trivially parallelizable tasks and doing it from cron.

The invocation looks like:

make -k -j 4 -l 3.99 -C [dir] [12 targets]

The trouble I'm running into is that sometimes one job will finish but the next one won't startup even though it shouldn't be stopped by the load average limiter. Each target takes about four hours to complete and I'm wondering if this might be part of the problem.

Edit: Sometimes a target does fail but I use the -k option to have the rest of the make still run. I haven't noticed any correlation with jobs failing and the next job not starting.

Answer 1

Does make think one of the targets is failing? If so, it will stop the make after the running jobs finish. You can use -k to tell it to continue even if an error occurs.

Answer 2

I'd drop the '-l'

If all you plan to run the the system is this build I think the -j 4 does what you want.

Based on my memory, if you have anything else running (crond?), that can push the load average over 4.

GNU make ref

Answer 3

@BCS

I'm 99.9% sure that the -l isn't causeing the problem because I can watch the load average on the machine and it drops down to about three and sometimes as low as one (!) without starting the next job.