At work, we need to build a jobs server for things like sending emails, building PDFs, crunching some data, etc. Obviously, we'd like to build on some sort of generic queueing system. I'm familiar with Gearman, and that this is exact problem that it tries to solve: putting jobs on a queue where workers come to pick them up. However, I'm seeing lots of mentions of Rabbitmq and am unclear how it's used in this scenario.
Is Rabbitmq a good framework to build a distributed jobs system on top of?
I would say that Gearman is better for queuing "jobs" and RabbitMQ is better for queuing "data". Of course, they are both really the same thing, but the way it works out for me is that if you are trying to "fan out" work to be done, and the workers can work independently, Gearman is the better way to do it. But if you are trying to feed data from a lot of sources down into fewer data consumers, RabbitMQ is the better solution.
The history of RabbitMQ, as something that allowed Twitter to take bursty loads of messages, and feed them into crusty old SMS gateways that could keep only one connection open, were rate limited, and didnt have retries, is illustrative of the kind of problems that RabbitMQ is good at solving.
It all depends what semantics you want to expose. It's really easy to do what Gearman does on top of RabbitMQ, which can certainly 'fan out' messages to independent workers.
But Gearman is built for purpose. IIUC, Gearman is a framework for processing jobs and not a messaging system as such. There are other such frameworks such as Celery that use RabbitMQ under the hood for that. Here is an article about Celery that is worth reading.
Hope this helps!
alexis