[RabbitMQ] Use it for JSON data transfer

Question

I am trying to use RabbitMQ for a distributed system that would work something like:

• a producer puts in a queue a JSON-formatted list of order ids
• several consumers pull out of that queue, do the business logic with that order ids and the result (JSON formatted) as well is put back into another queue
• from the second queue, another consumer will take the data and pass it back to the caller

I am still very new to RabbitMQ and I am wondering if this model is the right approach, given the fact that the data should be back as fast as possible (sometimes in the matter of seconds, max 5) so there are real time requirements. Also, how large can the message passed to a queue can be? The JSON that the producer will get back will be fairly large, based on what the consumer does.

Thanks for any ideas!

Answer 1

There's nothing wrong with the design you suggested.

The slight wrinkle is that enforcing "real time requirements" isn't straightforward. For instance, it's not currently possible to expire messages within a queue, so this would need to be handled by the clients when consuming messages.

The total size of messages in RabbitMQ <=1.8.1 was bounded by the amount of available RAM. As of 2.0.0, it's bounded by the amount of available disk space (i.e. rabbit will page messages to disk if it's running low on memory). Individual message sizes are recorded as 32-bit integers (IIRC), so individual messages cannot be larger than ~4GB; if this is a problem, consider saving the JSONs to network storage and passing some ID to them in the messages. Other than this, there aren't any constraints.

Answer 2

See page 47 in this presentation (InfoQ) for a great comparision between different messaging formats.