How can I get my very large program to link?

Question

Our next product has grown too large to link on a machine running 32-bit Windows. The sum total of all the lib files exceeds 2Gb and can only be linked on a 64-bit Windows machine. Eventually we will exceed that boundary, since our software tends to grow rather than contract and we are using a 32-bit linker (MS Visual Studio 2005): we expect to hit trouble when our lib size total exceeds 3Gb.

How can I reduce the size of the .lib files, or the .obj files without trimming code? For example, we use a lot of templates: is there any way of reducing their footprint? Is there any way of finding out what's causing the bloat from examining the .lib/.obj files? Can this be automated rather than inspected by eye? 2.5Gb is a lot of text to peer through and compare.

External constraints prevent us from shipping as anything other than a single .exe, so a DLL solution is not available.

Answer 1

OMFG!!!!! That's huuuuuge!

Apart from the fact I think it's too big to be rational... can't you use dynamic linking to avoid linking all the mess in compile time and only link in runtime what's necesary (I mean, loading dlls in demand)?

Answer 2

I had once been working on a project with several MLoC. While ours would still link on a 32bit machine, link times where abysmal and became a major problem, because developers were reduced to only get a dozen edit-compile-test cycles done per workday. (Compile times were handled pretty well by doing distributed compilation.)

We switched to dynamic linking. That increased startup time, but this could be managed by delay-loading of DLLs.

Answer 3

If it can't link on 32-bit, what are the chances it can run on 32-bit?

If it were my problem to solve, I'd start looking for large divisions in the program that could be handled by separate processes: break apart the program into modules based on the libraries that different sections of the program need.

If this is really hard to do, perhaps the library APIs could be changed / massaged / split to make it easier to split the program into multiple interacting processes, preferably along clearly understandable functional code divisions. (One process == one task.)

Answer 4

First, of course, make sure you compile with the 'Optimize for Size' option. If you do that, I wouldn't expect inlining, at least, to contribute significantly to the code size. The compiler makes a tradeoff for every inlining candidate regarding how much (if at all) it'd increase code size, compared to the performance boost it'd give. And if you're optimizing for size, the compiler won't risk bloating the code much. (Note that inlining very small functions can actually decrease code size)

Second, have you considered unity builds? That'd pretty much eliminate the linker entirely, and with only one translation unit, there'd be much less duplicate work and hopefully, a smaller memory footprint.

Finally, I know Visual Studio (or possibly the Windows SDK) has a 64-bit compiler (that is, a compiler that is itself a 64-bit application, not just a compiler producing 64-bit code). Consider using that. (I don't know if there is also a 64-bit linker)

I don't know i the linker is built with the LARGEADDRESSAWARE flag set. If so, running it on a 64-bit machine will let the process consume a full 4GB of memory instead of the 2 GB it normally gets. (if necessary, you can add the flag yourself by modifying the PE header)

Perhaps limiting the linkage of various symbols could help as well. If you know that a symbol won't be needed outside of the current translation unit, put it in an anonymous namespace. That might allow the compiler to trim down unused symbols before passing everything on to the linker

Answer 5

Does it need to be one big app?

One option is to split various modules into DLLs and load/unload them as needed.

Alternatively, you might be able to split into several apps and share data using mapped memory, pipes a DBMS or even simple data files.

Answer 6

I do not think there is any single tool that can give you statistics that you want/need. Using either .map files or the dumpbin utility with /SYMBOLS parameter plus some post-processing of the created log might help you get what you want.

If the statistics confirm your suspicion of template bloat, or even without the confirmation, it might be a good idea to do several things with the source:

1. Try using explicit instantiations and move template definitions into .cpp files. Of course this works only if you have limited and well known set of types/values that you use as arguments to the templates.
2. Add more abstraction and/or indirection. Factor code that does not depend on your template parameters into their own base classes or free functions. If you have several template type parameters, see if you cannot split the single class template into several base classes without overlapping template parameters. (See http://www2.research.att.com/~bs/SCARY.pdf.)
3. Try using the pimpl idiom; avoid instantiating templates in headers if you can, instantiate them only in .cpp files.
4. Templates are nice but sometimes ordinary classes work as well; e.g. avoid passing integer constants as non-type template parameters if you can pass them as parameter to ctor.

Answer 7

First of all, find out how to measure the size which is used by various features. Don't go ahead and try to play replace template usage or other things because you suspect that it makes a significant difference.

Run

dumpbin /HEADERS <somebinary>

to find out which sections in your binary are causing the huge size. Do you have a huge Debug Directory section? Strip symbols then. Is the Import Address Table large? Check the table and locate symbols which you don't need (a problem with templates is that symbols of template instantiations tend to be very very large). Similiar analysis can be done for the Exception Directory, COM Descriptor Directory etc..

Answer 8

I am highly skeptical that your actual code is 2GB; more likely you are compiling a lot of information. Consider offloading some of that information into a resource file, and embedding it in the executable as a separate step.

Answer 9

Try using the Symbol Sort program to show you where the main bits of bloat are in your code. Also just looking at the size of the raw .obj files will give you a reasonable idea of where to target.