For now I get a task_id from the async_result and have to save it the get it back later. Would be better if I knew what the task_id what made of so I can calculate it back instead of pulling from the DB. E.G: set a task with task_id=("%s-%s" % (user_id, datetime)). ...
Hi, I am running Django with Celery on top of RabbitMQ as a queue to handle some data processing tasks. I am kicking off celery tasks when a user first signs up, as well as periodically to update their data. However, I'd like to of course give priority to the tasks running users who are currently online. I noticed there was a priority se...
nohup python manage.py celeryd -f queue.log 2>queue.err 1>queue.out & Handles one request fine, then the client app posting the next job to the queues fails with this traceback. tasks.spawn_job.delay(details) File "/releases/env/lib/python2.6/site-packages/celery/task/base.py", line 321, in delay return self.apply_async(args, ...
I would like to be able to abort a task that is running from a Celery queue (using rabbitMQ). I call the task using task_id = AsyncBoot.apply_async(args=[name], name=name, connect_timeout=3) where AsyncBoot is a defined task. I can get the task ID (assuming that is the long string that apply_async returns) and store it in a database...
Background I have a web app that will create an image from user input. The image creation could take up to a couple seconds. Problem If I let the server thread, that is handling the request/response also generate the image, that is going to tie up a thread for a couple seconds, and possibly bog down my server, affect performance, kill...
I'm trying to use the AbortableTask feature of Celery but the documentation example doesn't seem to be working for me. The example given is: from celery.contrib.abortable import AbortableTask def MyLongRunningTask(AbortableTask): def run(self, **kwargs): logger = self.get_logger(**kwargs) results = [] for ...
I'm trying to create some Celery Periodic Tasks, and a few of them need to have the ability to change the run_every time at runtime. The Celery documentation says I should be able to do this by turning the run_every attribute into a property (http://packages.python.org/celery/faq.html#can-i-change-the-interval-of-a-periodic-task-at-runt...
I have a worker with concurrency set to 1 and a task that goes into infinite loop [for testing]. I submit a task T1 and the worker gets it and goes to work. I submit another task, it gets it and since its busy with T1 still, T2 goes into Queue [rabbitMQ]. I tried T2.revoke() and revoke(t2.task_id) And checked with the inspect to see th...
I`ve followed the guidelines in http://celeryq.org/docs/django-celery/getting-started/first-steps-with-django.html and created a view that calls my test method in tasks.py: import time from celery.decorators import task @task() def add(x, y): time.sleep(10) return x + y But if my add-method takes a long time to respond, how c...
I need to run some commands on my ubuntu box where my django project resides that requires root privileges, from within my django project. I'm using celery to fire off an asynch process, this process in turn calls shell commands that requires root privileges to succeed. How can I do this without risking creating huge security holes? P...
I've recently updated to celery 2.0 and I'm trying to use the new (to me) decorators for periodic tasks: @periodic_task(run_every=timedelta(days=1)) def some_task(**kwargs): #task here I also have some other code is my site that lists periodic tasks and how often they run using from celery.registry import tasks to get the list ...
I wanted to get your opinions on the merits of different AMQP implementations to be used with celery. I am looking particularly at message prioritization and job queue sizes. What are your thoughts? ...
I'm doing some largish scale data processing with nltk. In doing so we iterate through 2-3000 files sitting on the server, reading through them all, making changes, and closing them out. This is all handled via Celery tasks. The only problem is that we often find ourselves encountering this: IOError: [Errno 5] Input/output error A qui...
I am using celery for distributed task processing. I wanted to deploy my work on a web-host, just to show the working of my project. So how can i get djcelery to use a database (sqlalchemy) as backend with django instead of rabbitmq or other ampq servers. ...
I'm writing some software which will manage a few hundred small systems in “the field” over an intermittent 3G (or similar) connection. Home base will need to send jobs to the systems in the field (eg, “report on your status”, “update your software”, etc), and the systems in the field will need to send jobs back to the server (eg, “a fa...
I have below settings in my celery configuration file: CELERYD_LOG_DEBUG='FALSE' CELERYD_LOG_FILE=r'/var/log/celery/celeryd.log' CELERYD_LOG_LEVEL="ERROR" I looked at the settings file of the celery and there is no option to specify the log file size limit in the configuration. Even the code looks like its using the StreamHandler in...
I'm using celery with django and am trying to get a Task, like the one below: class task1 (Task) def run (self): launch_some_other_task.delay() But it doesn't seem to be working, I can go into more detail as far as my code but figured I would just ask first if this sort of thing will work as doesn't seem to be working for me....
I just installed and configured Celery with RabbitMQ for a Django project and I was having an issue running tasks when I imported them like so: from someapp.tasks import SomeTask It worked when I added the project name: from myproject.someapp.tasks import SomeTask I tried adding this into the settings.py file but it doesn't change ...
In development, it's a bit of a hassle to run the celeryd as well as the Django development server. Is it possible to, for example, ask celery to run tasks synchronously during development? Or something similar? ...
I want to develop an application which uses Django as Fronted and Celery to do background stuff. Now, sometimes Celery workers on different machines need database access to my django frontend machine (two different servers). They need to know some realtime stuff and to run the django-app with python manage.py celeryd they need acc...