Which persistent & lightweight queue messaging for cross domain (> 2) data exchange with rails integration ?

Question

Hi all,

I'm looking for the right messaging system for my needs. Can you help me ?

• For now, there won't be a huge amount of data to process, but I don't want to be limited later ...
• The machines are not just web servers, so the messaging tool should be lightweight, even if processing is not very speed.
• When some data change on a server, all servers should have the information and process it locally. (should I create one channel per server on each of them ?)
• The frontend is written on Rails, so it is important, in order to simplify the development, that there is a gem / plugin to manage communications and data sent.

At this time :

• RabbitMQ + workling seems to fit my needs. Could this be a right choice ?
• ActiveMQ make me afraid, because of Java (I really don't know very well Java, but it seems to me to be big CPU consumer)
• Others don't seem to be as mature as them.

There might be lot of development using this kind of technology, so I can't go to the wrong way !

Thank you for help.

Answer 1

I think you hit the 2 major products out there .

you can also check out these 2 products that would satisfy your needs somewhat:

http://memcachedb.org/memcacheq/

http://kr.github.com/beanstalkd/ - the advantage of this lib - it's in C, so it's fast, simple and it's been use in production for big apps - reducing response time. You can also nail it to any data exchange protocol you want in the background - soap, xml-rpc, json, since it all will be done asynchronously and you can incorporate your code, since the lib itself is very simple - check out example on the first page.

I can only talk about beanstalkd thou - since that's the only one I'm slightly familiar with - it was used in one of our projects. I can't comment for or against on other products.

Answer 2

I really like the beanstalkd protocol too btw - there is also Gearman too http://gearman.org/ Both beanstalkd and Gearman do synchronous sends to the broker - which is goodness - ActiveMQ does that too - but its optional. Lots of choices - but my advice would be to test the alternatives with your environment and see which one fits the best.