Executing a python script using subprocess.Popen() in a django view

Question

I've looked around a bit but I can't seem to solve this problem I have. I'd like to execute a python script within a view of my django app. I've placed the code I'd like to execute inside a django management command so it can be accessed via command line python manage.py command-name. I then tried to run this command using subprocess.Popen("python manage.py command-name",shell=True).

However, this command could take some time to execute so I'd like the view to continue and allow the script to execute in the background. Using subprocess.Popen alone seems to cause the view to hang until the script has finished, so I tried using a thread (following another SA question):

class SubprocessThread(threading.Thread):
def __init__(self, c):
    self.command = c
    self.stdout = None
    self.stderr = None
    threading.Thread.__init__(self)

def run(self):
    p = subprocess.Popen(self.command,
                         shell=True,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE)

    self.stdout, self.stderr = p.communicate()

and then executing it:

t = SubprocessThread("python manage.py command-name")
t.setDaemon(True)
t.start()
t.join()

However, the view still hangs: the cursor has a busy symbol and the AJAX on the page does not load. Otherwise the page's html seems to load fine and commands in the view after the thread call appear to finish normally (before the script finishes). Can someone please help me? I'd like the script to execute and do its own thing without holding up the view or AJAX calls on the page.

Answer 1

I wasted a lot of time trying to implement something similar, but had the same problems as you. Eventually, I gave up and implemented a beanstalk queue to handle the work.

http://kr.github.com/beanstalkd/

I put an id on the queue in the Django view, and then have a management command to run the consumer (watched by supervisord).

Using a queue means you could expand to multiple consumers, and allows to better manage the load (pausing the consumer if necessary without losing the work required).

Answer 2

Maybe you should use celery

Celery is a task queue/job queue based on distributed message passing. It is focused on real-time operation