Do you know/use any distributed job queue for python? Can you share links or tools
+1
A:
You probably want to look at multiprocessing's Queue. Included in Python 2.6, get it on PyPI for earlier versions of Python.
Standard library documentation: http://docs.python.org/library/multiprocessing.html On PyPI: http://pypi.python.org/pypi/multiprocessing
djc
2009-08-26 18:06:14
+2
A:
In addition to multiprocessing there's also the Celery project, if you're using Django.
Vinay Sajip
2009-08-26 18:06:35
Thanks for the link? Is it strictly used with django? can we use it for standard python projects?
Gopalakrishnan Subramani
2009-08-28 03:20:36
I don't see why not, with suitable adaptation (not sure how much work that'll be - depends on your exact requirement).
Vinay Sajip
2009-08-28 08:20:11
Celery has an underlying library, called Carrot, that you can use without Django.
Alex Gaynor
2010-01-08 03:56:50
Both Celery and Carrot works without Django. Or that is, you can use it from outside of a Django project. Recently someone even implemented paste support: http://bitbucket.org/twillis/celery-paste/
asksol
2010-02-01 16:25:03
+2
A:
There's also "bucker" by Sylvain Hellegouarch which you can find here:
It describes itself like this:
- bucker is a queue system that supports multiple storage for the queue (memcached, Amazon SQS for now) and is driven by XML messages sent over a TCP connections between a client and the queue server.
Michael Sparks
2009-08-26 18:10:01
+2
A:
there are a couple of python rabbitmq clients - see this thread for instance.
Martin DeMello
2009-08-26 20:45:57
+1
A:
redqueue? It's implemented in python+tornado framework, speaks memcached protocol and is optionally persistent into log files. Currently it is also able to behave like beanstalkd, the reserve/delete way in memcache protocol as well.
superisaac
2010-01-08 03:44:20
At present, I have been using celery. I will look at the redqueue. Thanks for answer
Gopalakrishnan Subramani
2010-01-08 15:03:32
A:
It's a year late or whatever, but this is something I've hacked together to make a queue of Processes executing them only X number at a time. http://github.com/goosemo/job_queue
Morgan
2010-07-21 20:04:24
+1
A:
Pyres is a resque clone built in python. Resque is used by Github as their message queue. Both use Redis as the queue backend and provide a web-based monitoring application.
optixx
2010-07-21 20:08:00