setting up a distributed computing network

Question

I need a lot of computing power for a new scientific application I built.

As I am on a budget, and since the app requires lots of CPU but little memory (400MB), a friend recommended that I buy many "old" machines (Pentium 4, 3Ghz, 1GB memory) and use them to run instances of my application (every app instance is independent of one another, and does a lot of computations and I\O).

There are some stores that seem to offer such machines for cheap ($100 a piece).

My question is, how hard would it be to install ubuntu linux on these machines, set it all up in a network and install things like python / cython on it?

how hard would it be to install condor, so that I can just 'submit jobs' to a central node and the computations will be carried out in the various nodes / units?

Answer 1

Have you considered using a cloud? Amazon's EC2 cloud might work out nicely for you. We've had success with it. You can create images that automatically boot with your OS and programs preinstalled.

Answer 2

I think you can setup one of such machine and take an image of the disk. Then, you can use that disk image on the rest of your cluster.

Though, I wonder how well this will work if your machines have different config.

Answer 3

Condor is exactly the right tool for this job, and it is relatively easy to set up Condor in straightforward configurations like this. You may want to look at something like rocksclusters.org for setting up the OS and the cluster of machines themselves.

Now, I would certainly benchmark these machines first. You may be surprised at how slow old machines are, and how your application performance is not directly related to clock speed. Certainly, the fastest, newest, highest core-count machines aren't going to be the most cost effective, but nether are the oldest, slowest one-way machines, either.