Distributed Computing in C++ ?

Question

Hi everyone,

I would like to know what tools, frameworks or libraries you would use, to spread your C++ application across multiple machines.
Im searching for a way to create a framework/environment, in which a master-server can hand many seperated Jobs to different seperated clients, who give back their result on a special Job. The results and jobs should be instances of "normal" C++ classes, so some sort of serialization would be nice.

It needs to be in C++ because i am from a scientific background and all frameworks i would like to use are written in it.

Answer 1

MPI is really cool! Info on Open MPI setup here, and there are some good tutorials here.

There's a simple example here: http://www.slac.stanford.edu/comp/unix/farm/mpi.html

Answer 2

codeproject.com has a nice tutorial on RCF, which simplifies thedefinition of remote methods, serialization of results and even network transport. You may want to give it a try.

But as it is C++, it doesn't give you the comfort of Java RMI (e.g. mobile code).

CORBA is also a possibility, but it's often "overkill" as it includes writing special interface definitions.

Answer 3

OpenMPI and/or OpenMP combinations work the best. We use OpenMPI on our supercomputing cluster to process large scientific jobs that require weeks of computing time.

As an additional note, MPI has C++ bindings from Boost::MPI which supports lovely stuff like serialization of STL types (valarray, vectors, strings, etc.) to allow easier message-passing on your part.

Answer 4

I used FlowVR to develop a distributed 3D flood simulation application.

Answer 5

Combine the above two answers - MPI to communicate across clusters, OpenMP to parallelise for cores on clusters. If you have graphics cards, throw CUDA etc into the mix too. That's what our distributed clusters do at work.

Edit: too slow.

Answer 6

You might investigate using CloudIQ Engine from Appistry. It allows you to distribute your C++ algorithms across any number of servers for processing. It also provides for process flow management for tasks. As part of the framework, failover is included, so if a task dies midstream (say someone pulls the plug on a machine), that task is automatically restarted on another node. And if that happens as part of a process flow, the whole flow does not have to be restarted, only the latest task. The framework automatically checkpoints your work at each step.