Is it possible to rent CPU cycles?

Question

I have an application that takes days to process data. Is there a service that would let me run my application on powerful computers?

I'm not running a website or a web service. This is taking lots and lots of data files, running them through a big custom application, and outputting a result.

It takes days on my PC and it's something that needs to be done every once in a while, but not continuously.

Cost isn't really an issue, in the sense that my company will pay for it, but of course it should be cheaper than buying a big-ass machine ourselves.

Answer 1

Have you considered Amazon EC2? You pay by the hour for what you use. No more, no less. You could event rent many servers at once to split the work load.

I'm not sure if that meets your requirement of "powerful computers", because they're just average servers, but at least it will give you a pay-as-you-go solution for running the program off of your own computer.

Answer 2

Amazon's EC2 Service is an excellent solution for your needs. You only pay for the time you use, and you can scale up to as many machines as you need.

From their information:

Elastic – Amazon EC2 enables you to increase or decrease capacity within minutes, not hours or days. You can commission one, hundreds or even thousands of server instances simultaneously. Of course, because this is all controlled with web service APIs, your application can automatically scale itself up and down depending on its needs.

Flexible – You have the choice of multiple instance types, operating systems, and software packages. Amazon EC2 allows you to select a configuration of memory, CPU, and instance storage that is optimal for your choice of operating system and application. For example, your choice of operating systems includes numerous Linux distributions, Microsoft Windows Server and OpenSolaris.

Answer 3

Hi Marlo,

Is your app memory intensive? From what you descibe it seems so.

We are going start a service of BIG remote SMP node for Memory Intensive runs.

The node has 512GB of RAM.

BIG memory is not enough, the system is also tuned for certain type of workloads which we define "dynamic" where multiple threads keeps exchanging data & synchronization. These type of scenarios typically congest the machine due to heavy communication.

If you application is memory intensive and dynamic in nature then you should consider our service.

Pierdamiano

Answer 4

If your application is not parallel, you won't get many advantages by running it in a "big machine", unless the bottleneck is in the virtual memory swapping. Even the Top500 supercomputers are not essentially faster than any PC for sequential workloads.

If your application can exploit parallelism maybe you could use your company's existent resources more efficiently than just deploying it in one and only pc. If you have a few dozens of computers, you could set up a loosely coupled heterogeneous cluster (or local grid, terminology changes with fashion).

Answer 5

EC2 is great, but do not forget that there are meanwhile other solutions available as well, such as http://www.rackspacecloud.com/ or GoGrid or MS Azure. Some of them cost less per CPU than EC2.

Lets keep the competition in this space alive ;)