Howto debug a (slow) linker on a debian system

Question

Hello, at my company we have a really annoying problem with our linker (ld 2.17). It links very very slow on a relatively fast system (Core Duo, 2GB Ram) and i don't really now how to fix this. It takes about five to ten minutes to compile a relatively big project (which takes about 5 seconds to link on my Gentoo system).

Personally i think it is a huge productivity killer, at least for me. We tried to use a more recent version of ld (2.19) but without any success. I asked in #debian on #freenode, but this problem seems to be very unique. I did not find any information about similar problems on the net. It only happens when we build with debug symbols. I changed the gcc debug-information flags to -g, -g3, and -ggdb, but that did not help either.

So my question is, how do you profile and debug a linker? I have never done anything like it, and i am unable to find any documentation about it. Basically any reasonable gprof gmon.out would be very helpful, as i could ask binutils developers about a concrete problem. I am just completely oblivious about this.

Best regards and thanks in advance, David

Edit: We 'fixed' our problem switching to debian lenny on most systems. Thanks for the answers!

Answer 1

If you're observing the slowdown running gcc (as opposed to directly running the linker as ld), try compiling with

$ gcc -save-temps -v [... rest of your command line ...]

This will print out all the intermediary commands, such as the internal collect2 and final ld, as well as ensure that the objects passed to those commands will remain on disk even after the command has completed.

Then you should be able to run the commands individually to find the worst stage, and then run it with different options or profiling.

For example,

$ echo 'int main() {}' > test.c
$ gcc -save-temps -v test.c
Using built-in specs.
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/paludis/sys-devel-gcc-4.3.3-r2/work/gcc-4.3.3/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/4.3.3 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.3.3 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.3.3/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.3.3/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/include/g++-v4 --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --disable-altivec --disable-fixed-point --enable-nls --without-included-gettext --with-system-zlib --disable-checking --disable-werror --enable-secureplt --enable-multilib --enable-libmudflap --disable-libssp --enable-libgomp --enable-cld --disable-libgcj --enable-objc-gc --enable-languages=c,c++,objc,obj-c++,treelang,fortran --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.3.3-r2 p1.1, pie-10.1.5'
Thread model: posix
gcc version 4.3.3 (Gentoo 4.3.3-r2 p1.1, pie-10.1.5)
COLLECT_GCC_OPTIONS='-save-temps' '-v' '-mtune=generic'
 /usr/libexec/gcc/x86_64-pc-linux-gnu/4.3.3/cc1 -E -quiet -v test.c -D_FORTIFY_SOURCE=2 -mtune=generic -fpch-preprocess -o test.i
ignoring nonexistent directory "/usr/local/include"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../x86_64-pc-linux-gnu/include"
#include "..." search starts here:
#include  search starts here:
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/include
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/include-fixed
 /usr/include
End of search list.
COLLECT_GCC_OPTIONS='-save-temps' '-v' '-mtune=generic'
 /usr/libexec/gcc/x86_64-pc-linux-gnu/4.3.3/cc1 -fpreprocessed test.i -quiet -dumpbase test.c -mtune=generic -auxbase test -version -o test.s
GNU C (Gentoo 4.3.3-r2 p1.1, pie-10.1.5) version 4.3.3 (x86_64-pc-linux-gnu)
        compiled by GNU C version 4.3.3, GMP version 4.2.4, MPFR version 2.4.1-p5.
warning: GMP header version 4.2.4 differs from library version 4.3.1.
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 20f3dbffbfd03e5311a257ae1239cd71
COLLECT_GCC_OPTIONS='-save-temps' '-v' '-mtune=generic'
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../x86_64-pc-linux-gnu/bin/as -V -Qy -o test.o test.s
GNU assembler version 2.19.1 (x86_64-pc-linux-gnu) using BFD version (GNU Binutils) 2.19.1
COMPILER_PATH=/usr/libexec/gcc/x86_64-pc-linux-gnu/4.3.3/:/usr/libexec/gcc/x86_64-pc-linux-gnu/4.3.3/:/usr/libexec/gcc/x86_64-pc-linux-gnu/:/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/:/usr/lib/gcc/x86_64-pc-linux-gnu/:/usr/libexec/gcc/x86_64-pc-linux-gnu/4.3.3/:/usr/libexec/gcc/x86_64-pc-linux-gnu/:/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/:/usr/lib/gcc/x86_64-pc-linux-gnu/:/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../x86_64-pc-linux-gnu/bin/
LIBRARY_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/:/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/:/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../x86_64-pc-linux-gnu/lib/:/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-save-temps' '-v' '-mtune=generic'
 /usr/libexec/gcc/x86_64-pc-linux-gnu/4.3.3/collect2 --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/crtbegin.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../.. test.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/crtend.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/crtn.o
$ ls
a.out  test.c  test.i  test.o  test.s
$ /usr/libexec/gcc/x86_64-pc-linux-gnu/4.3.3/collect2 --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/crtbegin.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../.. test.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/crtend.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/crtn.o
$ .../collect2 -v ...
collect2 version 4.3.3 (x86-64 Linux/ELF)
/usr/bin/ld -v --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/crtbegin.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../.. test.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/crtend.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.3/../../../../lib64/crtn.o
GNU ld (GNU Binutils) 2.19.1
$ 

If you need help building a debug version of ld, here's a quick recipe to get you going.

$ sudo apt-get install build-essential dpkg-dev
$ sudo apt-get build-dep binutils
$ apt-get source binutils
$ cd binutils-*
$ DEB_BUILD_OPTIONS='debug noopt nostrip' dpkg-buildpackage -uc -us
$ cd ..
$ sudo dpkg -i *.deb

Instead of that quick hack, though, I'd recommend using schroot or sbuild to avoid polluting your own system.

Answer 2

You can try gold (binutils-gold) instead of ld. It is supposed to be faster.

Here is a quote from Wikipedia Gold(linker)

The motivation for writing gold was to make a linker that is faster than the GNU linker[3], especially for large applications coded in C++.

The author of gold (Ian Lance Taylor) has published an (longish) article about linkers where he explains his motifs in writing gold and why most linkers are slow. If you are interested in the inner workings of linkers this article is worth reading.

Answer 3

To answer the profiling question; you should look at OProfile - this is a system-level profiler that can profile multiple running processes. It should allow you to identify which sub-process of the link is taking the most time, and furthermore will show which functions the most time is being spent.

Answer 4

I'd like to suggest two ways to check:

1. use strace to check what files linker is loading/parsing to link; with this you may know is there any unnecessary path is searched by linker.
2. use ld with -verbose option to know what ld is doing. Five minutes compare to five seconds should not be the linker's problem, it should be some problem of your host machine or some option.