Can message-oriented middleware be used instead of MPI to coordinate distributed computation?

Question

By message-oriented middleware I am referring to technologies such as Advanced Message Queuing Protocol.

Obviously AMQP is a different beast than MPI, but I would think distributed-memory computations that operate in a master-slave manner could be trivially implemented using AMQP, letting AMQP handle equitable work distribution to slaves as they finish pieces instead of managing the queue of work explicitly in the master.

The added benefit of AMQP (if you had thousands of machines working together) would be that the death of a single machine wouldn't stall progress of the computation at MPI_Bcasts, because AMQP could simply use a fanout instead of MPI_Bcast and that would be non-blocking to the progress of the overall computation.

Are there any examples of AMQP being used for task coordination in distributed computation?

Answer 1

I think it's helpful to distinguish between distributed computation and parallel computation. I take the view that parallel computation is a sub-class of distributed computation. In distributed computing many processors are used to tackle a problem, and the problem may be decomposed into a variety of tasks (eg client-server, to give a very simple example) and processors may be running a variety of codes.

In parallel computation, however, each processor is likely to be running the same code but getting a different part of the data to process.

Now, there is no hard and fast line where distributed computation ends and parallel begins, but if you look at the two ends of the spectrum there are canonical examples which have very different characteristics. I suppose Google might demonstrate a canonical example of distributed computation, while the kinds of scientific simulations that the large supercomputer centres run provide a canonical example of parallel computation.

All of the foregoing is simply background to my answer to your question:

Yes you could certainly use AMQP to tackle parallel computations and yes you could use MPI to implement distributed computations but I think you would be struggling against features of the protocols which are designed for the opposite ends of the spectrum.

And no, I don't know of anyone using AMQP for doing what I call parallel computation.