Message Queues in Ruby on Rails

Question

What message queues are people using for their Rails apps and what was the driving force behind the decision to choose it. Does the latest Twitter publicity over their in house queue Starling falling down affect any existing design decisions.

I am working on an app that will need a message queue to process some background tasks, I haven't done much of this, and most of the stuff I have seen in the past has been about Starling and Workling, and to be honest the application is not very big and this solution would probably suffice, but I'd love to get experience integrating the best solution possible as I'm sure I will integrate one into a bigger app at some point.

What message queues would you suggest for a Rails app???

EDIT: Thanks for the suggestions, I'm going to look at a few of them this weekend.

EDIT Again: I've had a look around and a little overwhelmed for choice. I am however going to go about integrating RabbitMQ with Workling into the app I am building, then if I ever need some knowledge about a fast queue then I will have this and know whether or not it fits my needs.

EDIT: Finding more and more that DJ suits me just fine, if I ever "outgrow" it on a site I'd say that Resque is where I would head.

Answer 1

Here are a few Ruby/Rails solutions, one or more of these may be a good fit depending on your needs:

http://xph.us/software/beanstalkd

http://rubyforge.org/forum/forum.php?forum_id=19781

http://backgroundrb.rubyforge.org

And, a hosted solution from Amazon which would make a great queue for sharing between Ruby/Rails and other components of a larger system:

http://aws.amazon.com/sqs

Hope this helps!

Answer 2

Chris Wanstrath from github was at the SF Ruby meetup recently, talking about their queue. They tried Starling, beanstalk, and some other variants before settling on Shopify's delayed_job. They are pretty aggressive with their use of backgrounding.

Here's a blog post from last year that talks about their move to DJ.

Where I am now we rolled our own several years ago, but I'm taking some ideas from DJ to improve the handling.

Answer 3

Rany Keddo gave a useful presentation about Starling + Workling at RailsConf Europe last year. He compared the different solutions available at the time.

Twitter's latest move away from Starling + Workling probably doesn't mean much to the regular rails app. They have a lot more issues of scale and probably have legacy issues with their datastore that prevents them from scaling past their current implementation.

Beanstalkd is a good alternative, simply because it runs as a daemon and has wrappers in other scripting languages (if you happen to change direction in the future or have different components written in other languages).

This link also has a good comparison of pros-cons of the various rails solutions available.

Answer 4

I use background_job which like delayed_job is a database-based queue.

A database makes an OK queue as long as you're not doing too much traffic in and out.

The reason I like background_job (and delayed_job) is that they do not require a separate process. They can run through cron. For me, this is of key importance because my messaging needs are even simpler than my meager sysadmin skills.

Answer 5

I would recommend delayed-job as a dead simple solution if you don't expect any heavy load. Pros: easy to setup, easy to monitor, simple code, doesn't have any external dependencies. Previously we used ActiveMessaging (with ActiveMQ and stomp), but it was an overkill for our project, so we switched to delayed_job for its simplicity.

Anyway, if you need very mature and fast solution, ActiveMQ is a very good choice. If you don't want to spend too much time on maintaining full-scale message queueing solution you don't really need, delayed_job is a way to go. Here's a good article about Scribd experience with ActiveMQ.

Answer 6

The Messaging Server you might want to go for is RabbitMQ. Erlang coolness, AMQP, good Ruby libs.

http://www.bestechvideos.com/2008/12/09/rabbitmq-an-open-source-messaging-broker-that-just-works

Answer 7

As an update -- GitHub have moved to Resque on Redis instead of Delayed job. However they still recommend delayed_job for smaller setups:

http://github.com/defunkt/resque