Open Source Queue that works with Java, PHP and Python

Question

I'm currently in the market for a new queue system for jobs we have in our system. I've tried beanstalk but it's been unable to keep up with the load. I'm looking for a simple system to get up and running that I can put pieces of data in from producers and have consumers in Java, PHP and Python pull data off and process it.

Ideally I'd like to see features such as:

Job verification -> jobs are removed from the queue only when I've marked them as finished (in case of failures I don't have to put the jobs back in)

Priorities -> ability to prioritize jobs

Multiple channels -> ability to have one queue that can service several apps with separate data streams(or databases)

Disk Persistence -> ability to have jobs written to disk in case of failures

anyone have any good suggestions?

Currently looking at RabbitMQ

Answer 1

How about Apache ActiveMQ.

Accessible from Java, PHP, Python.

Supports all the features you requested.

Answer 2

RabbitMQ is good messaging system and there are bindings for Java, PHP, Python and many other languages.

Answer 3

The Berkeley database can be used to build a priority queue with bindings to most relevant languages. The HA (High Availability) configuration can make it distributed as well. I believe the Sun Grid Engine, for instance, uses just that to synchronize jobs.