views:

88

answers:

2

I'm going to use a small example for reference. Consider a project with:

inner_definitions.o : inner_definitions.cpp inner_definitions.h
    gcc $^ -o $@

inner_class_1.o : inner_class_1.cpp inner_class_1.h inner_definitions.h
    gcc $^ -o $@    

inner_class_2.o : inner_class_2.cpp inner_class_2.h inner_definitions.h
    gcc $^ -o $@

outer_class.o : outer_class.cpp outer_class.h inner_class_1.h inner_class_2.h
    gcc $^ -o $@

executable.o : executable.cpp executable.h outer_class.h
    gcc $^ -o $@

executable : __?1__
    __?2__

But filling in the blanks __?1__ for the linker dependencies and __?2__ for the linker command isn't easy.

In this small example, one could argue that its easy to see that __?1__ = inner_definitions.o inner_class_1.o inner_class_2.o outer_class.o executable.o . However, this is clearly not a scalable solution as it forces each developer to understand all the dependencies of the code they are working with so they can figure out the dependencies by hand rather than by using the make utility.

Another solution would be to have a different variable for each object file that listed all its downstream dependencies: i.e __?1__ = executable.o $(executable_dependencies). This is not a desired solution because it forces the makefile to be compiled in the specific way so the variables are only used when they are fully defined. Also, for really large applications these variables might exceed the maximum variable length.

Yet another solution is to use archive .a files for linking. In this case, we could construct an inner_class_1.a that included both inner_defintions.o and inner_class_1.o, so it could be linked with any object file that needed inner_class_1.o without forcing the developer to reconstruct the dependencies. This approach seems promising, but involves having many duplicate files. Also, it doesn't appear that the gcc linker can handle nested archive files.

Is there another approach? What is the best approach? Can the gcc linker handle nested archive files?

A: 
Beta
So that step of "and grep around and discover that `foo::bar()` is defined in ..." is the step I want to make easy. The thought is that the developer who creates a new file would know what its immediate dependencies are, and would add those - make would figure out all the downstream dependencies and link them in as well. For reference, I'm actualy doing: http://www.xs4all.nl/~evbergen/nonrecursive-make.html
Alex Reece
Also, this involves passing *every* object file to the linker. Is this a scalable solution for large projects?
Alex Reece
"Passing extra object files to the linker makes no difference to the final executable" -- That's not true. All code from individual object files passed to the GNU linker goes into the executable. Unnecessary object files are only ignored if they're inside a static library.
slowdog
@Beta I believe that we're now on the same page about what the problem is. And your answer is that "[you] know of no robust way to automatically construct the object list" except by "following a simple naming convention". How would that work? I'm willing to try an advanced trick - I'm asking the question specifically to learn the trick.
Alex Reece
@slowdog: You're right, thank you, I didn't know that.
Beta
+2  A: 

The job you're trying to automate (picking the right object files to satisfy all references) is usually left to the linker, using static libraries (".a" files) to group the candidate object files, just as you suggest.

An important detail you may be missing: If you pass the linker an archive, it will only link in those files from the archive that are actually needed. So you can create archives at a fairly coarse level of granularity without necessarily bloating all your executables -- the linker will pick just what it needs -- although can easily end up with needlessly slow builds if you take this approach too far.

The GNU linker will not pull objects out of nested libraries. If you want to make one big library by merging many small ones, you can do that with the "addlib" command in an ar script. That will give you a single archive containing all of the object files without any nested library structure.

If the duplication of having .o files and .a files containing the same object code lying around bothers you, the fine manual describes a way to have make update the archives "directly".

slowdog
So what you're saying is that the linker usually handles this? I.e. it is standard and scalable to pass extra object files to the linker?One thing I'm concerned about is unit testing `inner_class1` - the unit test would only need to link `inner_class1.o` and `inner_definitions.o`. Is passing the complete list of object files to the linker for every unit test actually the appropriate thing to do?
Alex Reece
Also, the recursive .a files will also have lots of extra object code lying around (even without any .o files).
Alex Reece
Giving the linker static libraries containing more object files than are really needed is normal (try `ar t /usr/lib/libc.a` for example) and scales reasonably far. Choosing the right granularity of archives is a matter of trade-offs. For unit tests I would aim to give the linker less than "the complete list", so that unexpected dependencies on other modules result in linker errors rather than silently creeping in.
slowdog
"recursive .a files" sounds like you might be heading in a direction "considered harmful" http://miller.emu.id.au/pmiller/books/rmch/
slowdog
Another direction you might like to explore: You could split your application up into *dynamic* (aka "shared") libraries. These can refer to each other, so you need have no code duplication at all.
slowdog
@slowdog by "recursive .a files" I meant that I could make `outer_class.a` that contained `outer_class.o`, `inner_class1.a` and `inner_class2.a`. This fails because, as far as I'm aware, the linker doesn't recursively open archive files. The dynamic libraries still doesn't solve the problem of trying to figure out the downstream dependencies. Is there a way to avoid passing all the object files to the linker, either as a list of objects (generated by hand or computed) or in a massive `.a` file?
Alex Reece
As I said, nested archives won't work. Using "ar -M" with a script is the solution to that (edited the answer a little to make that clearer). Someone has to figure out the dependencies. If you pass everything to the linker, then the linker will do it for you. I would recommend you split your project into modules, specify the dependencies among modules manually, and pass all objects for the *modules* you know you need to the linker. Then the linker can worry about the individual object files. It's a compromise between micromanaging the code and building a monolith.
slowdog
Dynamic libraries do solve the "downstream dependency" problem: You can build a "libfoo.so", linking it against "libbar.so". If you then build an executable specifying only "libfoo.so", the linker will automatically pull in "libbar.so".
slowdog
Ooh, dynamic libraries sound like the way to go. Is there any drawback to have a separate library for each `.o` file with the same dependencies?
Alex Reece
Dynamic libraries are nice. Drawbacks: Deployment becomes annoying when your application consists of dozens of little libraries. Also, you may eventually find startup time becoming a problem, as dynamic libraries effectively defer a part of the linking process until runtime. Some discussion about that here: http://stackoverflow.com/questions/1410691/why-is-the-startup-of-an-app-on-linux-slower-when-using-shared-libs
slowdog
Using dynamic libraries doesn't solve the problem of unnecessary rebuilding.
Beta
@beta: I think make usually handle the unnecessary rebuilding ok, I'm more interested in the linking dependencies.@slowdog: So dynamic libraries don't actually solve the problem. The problem of "how do I convince `make` to figure out the dependencies and link in the all of the correct object files". They are one way to do it, but probably won't scale with many files ( because of the cost at runtime ) and don't actually solve the problem ( esp since sometimes there is a need to not use dynamic libraries ).
Alex Reece
The answer to "how do I convince make to figure out the dependencies and link in the all of the correct object files" is "you can't, that's not what make does".
slowdog
Ah, so thats the answer to this question.
Alex Reece