Is there any software available in linux which compiles a source code containing large number of files parallely on either multicore or distributed systems. Libraries like gcc or xserver takes very time for compilation on unicore/dual machine and most of the times it is frustrating when you need lot of recompilation. Is there any technique for compiling such source code parallely ?
+12
A:
On distributed-memory systems, you can use distcc to farm out compile jobs to other machines. This takes a little bit of setup, but it can really speed up your build if you happen to have some extra machines around.
On shared-memory multicore systems, you can just use make -j, which will try to spawn build jobs based on the dependencies in your makefiles. You can run like this:
$ make -j
which will impose no limit on the number of jobs spawned, or you can run with an integer parameter:
$ make -j8
which will limit the number of concurrent build jobs. Here, the limit is 8 concurrent jobs. Usually you want this to be something close to the number of cores on your system.
tgamblin
2009-12-27 17:33:12
AFAIK there are two schools on the "number of jobs". One being n+1 and the other 2n, where n is the number of cores. The rationale of both is that there is always a thread waiting once one gets completed... Personally, I have not seen real difference whether I put -j5 or -j8 on a quadcore
Kimvais
2009-12-27 18:58:06
I also haven't seen much benefit from either n+1 or 2n on multicores, but then I haven't seen much of a decrease in performance from those either. I'm suspicious that you don't get all the benefit you could out of those options b/c make doesn't set affinity for cores or sockets, but I haven't really tested this (I haven't even checked if make sets affinity, but I bet it doesn't).
tgamblin
2009-12-27 19:29:19
Building is often IO bound. Lots of small files to work on means moving a lot of data to and from the disk... Plus forks are inexpensive on linux, but they *do* cost something, which contributes to the non-scaling. Finally, recursive make and other poor makefile practices can steal *a lot*.
dmckee
2009-12-28 02:16:12
+3
A:
GNU make (the standard make installed on Linux systems) supports parallel command execution - take a look at http://www.gnu.org/software/make/manual/make.html#Parallel
anon
2009-12-27 17:33:12