Fast inter-process (inter-threaded) communications IPC on large multi-cpu system.

Question

What would be the fastest portable bi-directional communication mechanism for inter-process communication where threads from one application need to communicate to multiple threads in another application on the same computer, and the communicating threads can be on different physical CPUs).

I assume that it would involve a shared memory and a circular buffer and shared synchronization mechanisms.

But shared mutexes are very expensive (and there are limited number of them too) to synchronize when threads are running on different physical CPUs.

Answer 1

You probably want to start by looking at the existing libraries such as MPI and OpenMP. They tend to be tuned fairly well.

If you're willing to entertain more cutting-edge approaches, then you can try what Barrelfish is doing, see http://www.barrelfish.org/barrelfish_sosp09.pdf .

Answer 2

If you are going to use C++, boost has a portable pretty low level IPC library. It allows you to synchronize and share memory between processes.

http://www.boost.org/doc/libs/1_42_0/doc/html/interprocess.html