Rent A Cluster

Question

I am wondering if there are clusters available to rent.

Scenario:

We have a program that will take what we estimate a week to run(after optimization) on a given file. Quite possibly, longer. Unfortunately, we also need to do approximately 300+ different files, resulting in approximately 300 weeks of compute time(roundable to 6 wallclock years of continuously running job). For a research job that should be done - at the latest - by December, that's simply unacceptable. While we are exploring other options, I am investigating the option of simply renting a Beowulf cluster. The job is academic and will lead towards the completion of a PhD.

What would be ideal would be a company that we send the source and the job files to the company and then receive a week or two later the result files. Voila!

Quick googling doesn't turn up anything terribly promising.

Suggested Solutions?

Answer 1

Cloud computing sounds like what you need. Amazon, Microsoft and Google rent computer resources on a pay for what you use basis.

Amazon's service is the most mature, and there are several questions already about Amazon's service, EG here and here.

Answer 2

This lead may prove helpful:

http://lcic.org/vendors.html

And this is a fantastic resource site on the matter:

http://www.hpcwire.com

Answer 3

Amazon EC2 (Elastic Compute Cloud) sounds like exactly what you're looking for. You can sign up for one or more virtual machines (up to 20 automatically, more if you request permission), starting at $0.10 an hour per VM, plus bandwidth costs (free between EC 2 machines and Amazon's other web services). You can choose between several operating systems (various Linux distributions, OpenSolaris, Windows if you pay extra), and you can use pre-existing machine images or create your own. If you're using all open-source software and don't have much bandwidth costs, it sounds like it would cost you around $5000 to run your job (assuming that your 6 years of compute time was for something comparable to their small instances, with a single virtual CPU).

Once you sign up for the service and get their tools set up, it's pretty easy to get new virtual machines launched. I've even spent the $0.10 to launch a machine for a few minutes just to verify an answer I was giving someone here on StackOverflow; I wanted to check something on Solaris, so I just booted up an instance and had a Solaris VM at my disposal within 5 minutes.

Answer 4

I don't know where are you doing your PhD... Most of the Asian, European, and North American universities have some clusters. You can

• meet directly the people at the lab which is in charge of the cluster.
• ask your PhD director to arrange that. Maybe he/she have some friends that can handle that.

Also, the classical trick is to use the unused time of the computers of your lab/university... Basically, each computer run a client application that crunch numbers when the computer is not used. See http://boinc.berkeley.edu/

Answer 5

One option is to rent the virtual resources equivalent of whatever number of PCs you need, and set them up as a cluster, using the Amazon Elastic Compute Cloud.

Setting up a beowulf cluster of those is entirely possible.

Check out this link which provides resources and software to do exactly that.

Answer 6

The thread has been replete with pointers to Amazon's EC2 - and correctly so. They are the most mature in this area. Recently, they've released their elastic map-reduce platform which sound similar (although not exactly) like what you are trying to do. Google is not an option for you as their compute model doesn't support the generic compute model you need.

Answer 7

• For academic/scientific use, there are several public centers offering HPC capability. In Europe, there is DEISA. http://www.deisa.eu/ and DEISA members. There must be similar possibilities in the US, probably thru the NSF.

• For commercial use, check IBM Deep Computing On Demand offerings. http://www-03.ibm.com/systems/deepcomputing/cod/