How can I write a makefile to auto-detect and parallelize the build with GNU Make?

Question

Not sure if this is possible in one Makefile alone, but I was hoping to write a Makefile in a way such that trying to build any target in the file auto-magically detects the number of processors on the current system and builds the target in parallel for the number of processors.

Something like the below "pseudo-code" examples, but much cleaner?

all:
    @make -j$(NUM_PROCESSORS) all

Or:

all: .inparallel
    ... build all here ...

.inparallel:
    @make -j$(NUM_PROCESSORS) $(ORIGINAL_TARGET)

In both cases, all you would have to type is:

% make all

Hopefully that makes sense.

UPDATE: Still hoping for an example Makefile for the above. Not really interested in finding the number of processes, but interested in how to write a makefile to build in parallel without the -j command line option.

Answer 1

The detection part is going to be OS dependent. Here's a fragment that will work on Linux and Mac OS X:

NPROCS:=1
OS:=$(shell uname -s)

ifeq($(OS),Linux)
  NPROCS:=$(shell grep -c ^processor /proc/cpuinfo)
endif
ifeq($(OS),Darwin) # Assume Mac Os X
  NPROCS:=$(shell system_profiler | awk '/Number Of CPUs/{print $4}{next;}'}
endif

To get it to go you are probably going to have to re-invoke make. Then you problem is preventing infinite recursion. You could manage that by having two makefiles (the first of which only resets the -j value), but it is probably possible to finesse it.

Answer 2

After poking around the LDD3 chapter 2 a bit and reading dmckee's answer, I came up with this not so great answer of using two makefiles (I would prefer just one).

$ cat Makefile
MAKEFLAGS += -rR --no-print-directory

NPROCS := 1
OS := $(shell uname)
export NPROCS

ifeq ($J,)

ifeq ($(OS),Linux)
  NPROCS := $(shell grep -c ^processor /proc/cpuinfo)
else ifeq ($(OS),Darwin)
  NPROCS := $(shell system_profiler | awk '/Number of CPUs/ {print $$4}{next;}')
endif # $(OS)

else
  NPROCS := $J
endif # $J

all:
    @echo "running $(NPROCS) jobs..."
    @$(MAKE) -j$(NPROCS) -f Makefile.goals $@

%:
    @echo "building in $(NPROCS) jobs..."
    @$(MAKE) -j$(NPROCS) -f Makefile.goals $@
$ cat Makefile.goals
MAKEFLAGS += -rR --no-print-directory
NPROCS ?= 1

all: subgoal
    @echo "$(MAKELEVEL) nprocs = $(NPROCS)"

subgoal:
    @echo "$(MAKELEVEL) subgoal"

What do you think about this solution?

Benefits I see is that people still type make to build. So there isn't some "driver" script that does the NPROCS and make -j$(NPROCS) work which people will have to know instead of typing make.

Downside is that you'll have to explicitly use make -f Makefile.goals in order to do a serial build. And I'm not sure how to solve this problem...

UPDATED: added $J to above code segment. Seems work work quite well. Even though its two makefiles instead of one, its still quite seamless and useful.

Answer 3

I'll skip over the $(NPROCS) detection stuff, but here's how you could do this in a single Makefile (this is probably GNU Make specific, but that looks like the variant you're running):

ifeq ($(NPROCS),)
# Code to set NPROCS...
%:
    $(MAKE) -j$(NPROCS) NPROCS=$(NPROCS)
else
# All of your targets...
endif

See Defining Last-Resort Default Rules and Overriding Part of Another Makefile in the GNU Make Manual.