Would a cloud-based compiler be feasible?

Question

Would it be any practical benefit to write a cloud-based compiler, that would spread compiled units of code on different machines in the cloud? Could there be a benefit from obtaining a software-as-a-service architecture right within the app just after compiling, or would the inherent latency make such an approach impractical?

Answer 1

It depends. We used to have old C++ based compiles that took 3-4 hours. For something like that, it would be very useful to offload the compile. But in C++ projects, this is often even less possible.

With C# or Java, the compile times are significantly faster, so it may not be as important.

Answer 2

I believe it would be impractical. Today's hardware can do any compilation of small and medium-sized projects in reasonable time; more so with multi-core CPUs.

The only exception is building an entire operating system (e.g. Debian) from sources; for such application, build farms are in widespread use. However, users in need of build farms can typically create them themselves, and don't need to go to the cloud.

Answer 3

I've used such a system, but it worked on a local cluster, not a cloud. But the principle would be exactly the same. Unfortunately I can't remember what it was called - but it was cool, watching your source files get farmed out to the other PCs in your department.

Edit: It was called IncrediBuild.

Answer 4

Xcode has a distributed build feature that lets you do this, but I think that on anything other than a LAN it would be very slow most of the time.

Answer 5

I think this could be useful if it were some kind of continuous integration tool. For some development environments, getting the proper setup is difficult. For example, I once worked on a project where the team compiled SWF files using FlashDevelop and Adobe's Flex SDK. It was sometimes a pain to set up on an individual's computer. If a service could monitor a project directory or git/SVN/etc. repository and build the latest SWF, however, I think that would have been useful.

I imagine this would have a use for cross-platform projects too. Or, this service could support an in-browser IDE like Mozilla's Bespin.

Answer 6

You can use distcc and make -j for distributed compilation of most typical unix code. If you regularly compile big chunks of code, it might get you big speedups... afaik samba (free smb implementation) developers use it for this. distcc does only the compilation phase in a distributed way, leaving preprocessing and linking to the master machine.

Interaction with "the cloud" might induce latency, but I still think with more complicated c++ code it might be very useful. I guess if you have more than 100 compilation units (f.e. .cpp files), you could get noticeable speedup.

Answer 7

I've seen a nice demo of running JUnit tests on top of GridGain, and GridGain can be run in the cloud.

I don't see much value in the compiler being in the cloud though.

Answer 8

I'm not sure if I've misunderstood your point or if the other answers have. Are you talking about some sort of automatic parallelisation task? The answers given so far appear to be talking about distributed compilation - i.e. using a cloud to speed up compilation times. I assumed you were instead talking about a compiler that targets cloud computing resources.

If you were in fact talking about distributed compilation, then obviously things like distcc will do what you need.

If you were asking the much more interesting (IMHO) question about whether a compiler that targets distributed architectures would be useful, my answer is a resounding 'yes'. However, feasibility is at the heart of the problem. Latency is not the problem as such, however coherence (i.e. ensuring that the correct versions of all the units are in place) and having decent heuristics would be an issue.

The best place to look would probably be the Occam programming language - it targeted the transputer, which was not entirely dissimilar to the kinds of distributed systems architectures we're interested in these days. I believe there is some work that follows on from Occam that might provide useful clues as to what the state of the art is.