How to Setup a Low cost cluster

Question

At my house I have about 10 computers all different processors and speeds (all x86 compatible). I would like to cluster these. I have looked at openMosix but since they stopped development on it I am deciding against using it. I would prefer to use the latest or next to latest version of a mainstream distribution of Linux (Suse 11, Suse 10.3, Fedora 9 etc).

Does anyone know any good sites (or books) that explain how to get a cluster up and running using free open source applications that are common on most mainstream distributions?

I would like a load balancing cluster for custom software I would be writing. I can not use something like Folding@home because I need constant contact with every part of the application. For example if I was running a simulation and one computer was controlling where rain was falling, and another controlling what my herbivores are doing in the simulation.

Answer 1

I recently set up an OpenMPI cluster using Ubuntu. Some existing write up is at https://wiki.ubuntu.com/MpichCluster .

Answer 2

Cluster it for what?
If you want a web-cluster you just need to install your favorite distro with the software and a load balancer.

Answer 3

You only need a cluster if you know what you want to do. Come back with an actual requirement, and someone will suggest a solution.

Answer 4

Your question is too vague. What cluster application do you want to use?

By far the easiest way to set up a "cluster" is to install Folding@Home on each of your machines. But I doubt that's really what you're asking for.

I have set up clusters for music/video transcoding using simple bash scripts and ssh shared keys before.

I manage mail server clusters at work.

Answer 5

I think he's looking for something similar with openMosix, some kind of a general cluster on top of which any application can run distributed among the nodes. AFAIK there's nothing like that available. MPI based clusters are the closest thing you can get, but I think you can only run MPI applications on them.

Answer 6

Linux Virtual Server

http://www.linuxvirtualserver.org/

Answer 7

Take a look at Rocks. It's a fullblown cluster "distribution" based on CentOS 5.1. It installs all you need (libs, applications and tools) to run a cluster and is dead simple to install and use. You do all the tweaking and configuration on the master node and it helps you with kickstarting all your other nodes. I've recently been installing a 1200+ nodes (over 10.000 cores!) cluster with it! And would not hesitate to install it on a 4 node cluster since the workload to install the master is none!

You could either run applications written for cluster libs such as MPI or PVM or you could use the queue system (Sun Grid Engine) to distribute any type of jobs. Or distcc to compile code of choice on all nodes!

And it's open source, gpl, free, everything that you like!

Answer 8

I use pvm and it works. But even with a nice ssh setup, allowing for login without entering passwd to the machine, you can easily remotely launch commands on your different computing nodes.