How to best launch an asynchronous job request in Django view?

Question

One of my view functions is a very long processing job and clearly needs to be handled differently. Instead of making the user wait for long time, it would be best if I were able to lunch the processing job which would email the results, and without waiting for completion notify the user that their request is being processed and let them browse on. I know I can use os.fork, but I was wondering if there is a 'right way' in terms of Django. Perhaps I can return the HTTP response, and than go on with this job somehow?

Answer 1

Why not simply start a thread to do the processing and then go on to send the response?

Answer 2

There are a couple of solutions to this problem, and the best one depends a bit on how heavy your workload will be.

If you have a light workload you can use the approach used by django-mailer which is to define a "jobs" model, save new jobs into the database, then have cron run a stand-alone script every so often to process the jobs stored in the database (deleting them when done). You can use something like django-chronograph to manage the job scheduling easier

If you need help understanding how to write a script to process the job see James Bennett's article Standalone Django Scripts for help.

If you have a very high workload, meaning you'll need more than a single server to process the jobs, then you want to use a real distribute task queue. There is a lot of competition here so I can't really detail all the options, but a good one to use with for Django apps is celery.

Answer 3

Before you select a solution, you need to determine how the process will be run. I.e is it the same process for every single user, the data is the same and can be scheduled regularly? or does each user request something and the results are slightly different ?

As an example, if the data will be the same for every single user and can be run on a schedule you could use cron.

See: http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/ or http://docs.djangoproject.com/en/dev/howto/custom-management-commands/

However if the requests will be adhoc and you need something scalable that can handle high load and is asynchronous: what you are actually looking for is a message queing system. Your view will add a request to the queue which will then get acted upon.

There are a few options to implement this in Django: Django Queue service is purely django & python and simple, though the last commit was in April and it seems the project has been abandoned. http://code.google.com/p/django-queue-service/

The second option if you need something that scales, is distributed and makes use of open source message queing servers: celery is what you need

http://ask.github.com/celery/introduction.html http://github.com/ask/celery/tree

Answer 4

I'm not very sure but there have been instances when I stopped the development server and it raised an interrupt in a select() call. Isn't that asynchronous behaviour on its part? Please correct me, or if possible, point me to the relevant documentation? Thanks.