High-performance message passing from C to Erlang

Question

I have a C server (a data feed handler) that has the potential to send millions of tiny messages per second across a few thousand long-lived Erlang processes. In a single day, some of these processes will receive a few thousand messages, while others will receive tens of millions. My interests are threefold:

1. to minimize latency — shortening the length of time from when the C server sends a message to when the Erlang process receives it. A lot of work can be done on things that happen before the message is sent or after the message is received, but this question is about the link between C and Erlang.

2. to maximize throughput — the faster the better; if the C server can send out 10% more messages per second across all of the Erlang processes, that's a big win.

3. to maximize predictability — if latency or throughput can suddenly degrade by several orders of magnitude due to something like network congestion, that's a big fail.

How can I send messages from a C program to an Erlang process in a high-performance way? What tricks can I do to minimize latency, maximize throughput, or make the communication between Erland and C more predictable? There are many aspects to this. We can choose the protocol for communication between an Erlang program and other programs. We can use Distributed Erlang with a "C Node" for communication between Erlang and C. We can use ports or linked-in drivers.

Answer 1

Use RabbitMQ?

You may need a lighter weight solution, but since there is a messaging middleware component already written in Erlang and implementing AMQP, it should be easy to try out RabbitMQ and see how it goes...

Answer 2

I would suggest you invert your system, embed your C code to Erlang VM as linked-in driver. IMHO it is way how you can achieve fastest solution in manageable way.