Queue System Farm questions

views:

answers:

Queue System Farm questions

Hi,

Until now, we haven't really needed to implement a queue system in our infrastructure, but now we need some features that cannot be done thru cron jobs (too slow).

Our infrastructure is LAMP (PHP) with some NoSQL.

I have looked at Gearman and it seems to be perfect for our needs. I know Digg uses it, as well as Yahoo so it must be reliable.

Is there any management tools that will automatically start a worker when it dies, or when new tasks are created?
How do you handle multiple projects under one same infrastructure (project A has 5 tasks, project B has 10, project C has 1)
Is there a way to do periodic tasks easily?

I've seen Celery and it looks good but It seems like it's only for AMQP (Or for RabbbitMQ only) but it's in python, and I haven't found much info on whether you can use non-python tasks.

Thank you

Celery supports webhook tasks, which enables it to call out to other languages via HTTP:

http://celeryq.org/docs/userguide/remote-tasks.html

Celery also supports Redis, MongoDB, MySQL/PostgreSQL/Oracle and beanstalkd: http://celeryq.org/docs/tutorials/otherqueues.html

RabbitMQ is the most reliable choice though.

asksol 2010-08-27 08:55:13

Yeah, but the idea of a queue system is to scale, if you use HTTP for it, then i could just not use a queue system and run it as "web" scripts.My choice for Gearman instead of RabbitMQ is because I know reddit.com has ran rabbitmq with lots of problems, and Yahoo runs Gearman doing 6M messages a day

angelito 2010-08-28 19:23:16

You don't understand. It's still a "queueing system", it just uses HTTP callbacks instead of native functions. Say you want to import some contacts, then you expose an url "http://mysite/user/contacts/import", and the queueing system calls that URL later.6 million messages/day is not that much.

asksol 2010-08-30 08:45:15

ansaurus

tags:

views:

answers:

Queue System Farm questions

related questions