How can I efficiently build different versions of a component with one Makefile

Question

I hope I haven't painted myself into a corner. I've gotten what seems to be most of the way through implementing a Makefile and I can't get the last bit to work. I hope someone here can suggest a technique to do what I'm trying to do.

I have what I'll call "bills of materials" in version controlled files in a source repository and I build something like:

make VER=x

I want my Makefile to use $(VER) as a tag to retrieve a BOM from the repository, generate a dependency file to include in the Makefile, rescan including that dependency, and then build the product.

More generally, my Makefile may have several targets -- A, B, C, etc. -- and I can build different versions of each so I might do:

make A VER=x
make B VER=y
make C VER=z

and the dependency file includes information about all three targets.

However, creating the dependency file is somewhat expensive so if I do:

make A VER=x
...make source (not BOM) changes...
make A VER=x

I'd really like the Makefile to not regenerate the dependency. And just to make things as complicated as possible, I might do:

make A VER=x
.. change version x of A's BOM and check it in
make A VER=x

so I need to regenerate the dependency on the second build.

The check out messes up the timestamps used to regenerate the dependencies so I think I need a way for the dependency file to depend not on the BOM but on some indication that the BOM changed.

What I've come to is making the BOM checkout happen in a .PHONY target (so it always gets checked out) and keeping track of the contents of the last checkout in a ".sig" file (if the signature file is missing or the contents are different than the signature of the new file, then the BOM changed), and having the dependency generation depend on the signatures). At the top of my Makefile, I have some setup:

BOMS = $(addsuffix .bom,$(MAKECMDGOALS)
SIGS = $(subst .bom,.sig,$(BOMS))

DEP = include.d
-include $(DEP)

And it seems I always need to do:

.PHONY: $(BOMS)

$(BOMS):
     ...checkout TAG=$(VER) $@

But, as noted above, if i do just that, and continue:

$(DEP) : $(BOMS)
     ... recreate dependency

Then the dependency gets updated every time I invoke make. So I try:

$(DEP) : $(SIGS)
     ... recreate dependency

and

$(BOMS):
     ...checkout TAG=$(VER) $@
     ...if $(subst .bom,.sig,$@) doesn't exist
     ...  create signature file
     ...else
     ...  if new signature is different from file contents
     ...    update signature file
     ...  endif
     ...endif

But the dependency generation doesn't get tripped when the signature changes. I think it's because because $(SIGS) isn't a target, so make doesn't notice when the $(BOMS) rule updates a signature.

I tried creating a .sig:.bom rule and managing the timestamps of the checked out BOM with touch but that didn't work.

Someone suggested something like:

$(DEP) : $(SIGS)
    ... recreate dependency
$(BOMS) : $(SIGS)
    ...checkout TAG=$(VER) $@
$(SIGS) :
    ...if $(subst .bom,.sig,$(BOMS)) doesn't exist
    ...  create it
    ...else
    ...  if new signature is different from file contents
    ...    update signature file
    ...  endif
    ...endif

but how can the BOM depend on the SIG when the SIG is created from the BOM? As I read that it says, "create the SIG from the BOM and if the SIG is newer than the BOM then checkout the BOM". How do I bootstrap that process? Where does the first BOM come from?

Answer 1

I'm not a make expert, but I would try have $(BOMS) depend on $(SIGS), and making the $(SIGS) target execute the if/else rules that you currently have under the $(BOMS) target.

$(DEP) : $(SIGS)
    ... recreate dependency
$(BOMS) : $(SIGS)
    ...checkout TAG=$(VER) $@
$(SIGS) :
    ...if $(subst .bom,.sig,$(BOMS)) doesn't exist
    ...  create it
    ...else
    ...  if new signature is different from file contents
    ...    update signature file
    ...  endif
    ...endif

EDIT: You're right, of course, you can't have $(BOM) depend on $(SIGS). But in order to have $(DEP) recreate, you need to have $(SIG) as a target. Maybe have an intermediate target that depends on both $(BOM) and $(SIG).

$(DEP) : $(SIGS)
    ... recreate dependency
$(NEWTARGET) : $(BOMS) $(SIGS)
$(BOMS) : 
    ...checkout TAG=$(VER) $@
$(SIGS) :
    ...if $(subst .bom,.sig,$(BOMS)) doesn't exist
    ...  create it
    ...else
    ...  if new signature is different from file contents
    ...    update signature file
    ...  endif
    ...endif

$(SIGS) might also need to depend on $(BOMS), I would play with that and see.

Answer 2

Make is very bad at being able to detect actual file changes, as opposed to just updated timestamps.

It sounds to me that the root of the problem is that the bom-checkout always modifies the timestamp of the bom, causing the dependencies to be regenerated. I would probably try to solve this problem instead -- try to checkout the bom without messing up the timestamp. A wrapper script around the checkout tool might do the trick; first checkout the bom to a temporary file, compare it to the already checked out version, and replace it only if the new one is different.

If you're not strictly bound to using make, there are other tools which are much better at detecting actual file changes (SCons, for example).