Multicore make in parallel rather than "threaded" execution?

Question

I have a Makefile to execute a test suite that looks roughly like this:

%.diff.png: %.test.png
    echo '$*: Comparing with good PNG.'

%.test.png: %.pdf
    echo '$*: Converting PDF to PNG.'

%.pdf: %.tex
    echo '$*: Generating PDF output.'

with all the echo statements supplemented with the actual tests in the real Makefile.

When I execute all of these tests with make test (the test target is not shown above), I obviously get linear output:

...
umtest200b: Generating PDF output.
umtest200b: Converting PDF to PNG.
umtest200b: Comparing with good PNG.
...

When I run these tests with multi-job make (make -j2 test, say), the tests are executed in a "threaded" order:

...
umtest202a: Generating PDF output.
umtest202b: Generating PDF output.
...
umtest202a: Converting PDF to PNG.
umtest202b: Converting PDF to PNG.
...
umtest202a: Comparing with good PNG.
umtest202b: Comparing with good PNG.
...

Perhaps you can see the problem; before discovering if the test fails it's already run through all of the PDF generations and PNG conversions for every single other test.

Is there a way to organise the tests so that even when run with multiple jobs the tests are run to completion before moving on to the next test? Or is this a task for a better make?

Answer 1

I did some reading on the -j flag and I think the issue is that this starts new jobs for independent rules. The jobs for every image are independent, so the first dependency for each target will be run like you're seeing.

I think the main problem here is that this task is inherently serial, that's to say that the dependencies for each target must be run in order, one after another. So within each target, you don't have a way to take advantage of multiprogramming, at this level of granularity.

As make doesn't seem to be processing dependencies in the order that you want, perhaps a unit testing framework that supports parallel tests would be a better fit. I know through some Googling that several of these exits, but I can't recommend when as I haven't used any and am not sure which language you're using.

Answer 2

Unless the order of execution of targets is explicitly specified, make executes them in an arbitrary order--both in parallel and in non-parallel mode. This order undergoes internal algorithms of make and in your case yields unpleasant results.

The solution is to impel an explicit ordering to your tests.

From your part of makefile, I assume that there exists an unrevealed "source" variable that contains a list of targets to test (on these targets implicit ones depend). Also immediate expansion of that variable yields correct results.

Assume the variable's like that:

list=file1 file2 file3 ... fileN

then, to solve the problem, it would suffice to generate the following dependencies:

file2: file1
file3: file2 
...
fileN: fileN-1
run_tests: fileN

How should we generate it? Let's write an foreach-eval loop after the list variable gets its value:

len=$(words $(list))
$(foreach t,                                                  \
   $(join                                                     \ 
      $(addsuffix : ,$(wordlist 2,$(len),$(list)) run_tests), \
      $(list)                                                 \
   )                                                          \
   , $(eval $(t))                                             \
)

This will generate a part of makefile just like C preprocessor does (or, more correctly, LISP macros). It will work like this: for each item t in the list formed by joining (concatenating each element of first list with corresponding element of the second) the list file2 file3 ... fileN run_tests, to each element of which a suffix : is added (thus forming file2: file3: ... fileN: run_tests:), with source list file1 file2 ... fileN; - for each such item t in list with joined elements evaluate it as part of makefile's source, thus acting as the rules shown above were prescribed.

Now you're only to invoke run_tests target and it will go one-by-one.

Answer 3

Suddenly, another idea came into my mind!

%.diff.png:
    $(MAKE) $*.test.png
    echo '$*: Comparing with good PNG.'

This solves everything except that tests may not be invoked in proper order (but in practice they most likely will go well).