How to generate one large dependency map for the whole project that builds with makefiles?

Question

I have a gigantic project that is built using makefiles. Running make at the root of the project takes over 20 minutes when no files have changed (i.e. just traversing the project and checking for updated files).

I'd like to create a dependency map that will tell me which directories I need to run 'make' in based on the file(s) changed. I already have a list of updated files that I can get from my version control system, and I'd like to skip the 20 minutes of traversing and get straight to the locations that do need to be recompiled.

The project has a mix of several languages and custom tools, so this would ideally be language-independent (i.e. it would only process all makefiles to generate dependencies). I'll settle for a C/C++-specific solution, too, as the majority of the project is in C++. The project is built on Linux.

[Edit - clarifications, responses to comments]

The project truly is gigantic, it's taking 20 minutes simply because of the size and because of all things described in the link below ("recursive makes consider harmful"). Unfortunately, the project is put together from many pieces coming from many different places, and there's little flexibility in what I can do with low-level Makefiles... the top-level ones putting the pieces together are the ones I have control over.

Answer 1

If a simple make with no updates takes 20 minutes, that must be a huge project! Considering that your makefile configuration is OK you could try something like ccache.

notice that even if you get the list of files you just need to comiple, you might still need to run make from the root of the project because the changes will have to be linked back into the main binaries. So if ccache is not good enough yet, and you insist in solving that problem yourself, you will still have to track which binary files you have to re-link again with "ld"

Answer 2

Hi,

One thing you could look for is "makefile graph" -- there are a few projects out there to make dependency trees from large makefile projects. Here's one: http://sailhome.cs.queensu.ca/~bram/makao/index.html . I think it will both create a pretty graph, and actually build things.

If you want to roll your own Makefile parser, and essentially replace make with your own custom script, my advice would be "don't". But there is a collection of Perl modules: Makefile::Parser. It claims to "pass 51% of the GNU Make test suite".

If you want to just look at what's taking so long, you can turn on Make's debugging output. Bits of it are usually sort of useless, like the pages and pages of Make deciding whether or not to apply implicit rules, but maybe there will be some obvious thing that happens that doesn't need to happen. If you want to do this, you could look at remake, "GNU Make with comprehensible tracing and a debugger".