Best Work Queue service for distributed clusters

Question

Hi there.

I require a simple work queue type system for asynchronous task management.

I have looked at both beanstalkd and gearman. However, both these seem to assume that the client and the queue server are on the same network, and therefore that there will always be a reliable network between them.

I need one that can support the client and server being in different places in the world, and be able to manage temporary loss of network connection between clusters.

Ideally, this would work in such a way where I post a job to a local proxy that attempts to send it to the main queue server. If there is no network connection, it would try again later, however it would not lose the job or delay the client.

Any recommendations?

Answer 1

Try ActiveMQ. Client libraries (no matter if sending or consuming messages) are smart enough to automatically reconnect on network outages, server reboots etc. For me, it works nicely on my LAN and with a few machines connected over VPN. I use Apache NMS as client library for .NET and single ActiveMQ broker instance.

It features also HTTP support, however I haven't tried that yet, don't know if reconnect support etc. is the same.

There is also support for running more than one broker (server) for clustering and load balancing, called "network of brokers". Brokers can even sit behind a firewall. Could be an option for you if you have a large number of producers/consumers in more than one place.