How do I run another script in Python without waiting for it to finish?

Question

I am creating a little dashboard for a user that will allow him to run specific jobs. I am using Django so I want him to be able to click a link to start the job and then return the page back to him with a message that the job is running. The results of the job will be emailed to him later.

I believe I am supposed to use subprocess.Popen but I'm not sure of that. So in pseudocode, here is what I want to do:

if job == 1:
    run script in background: /path/to/script.py
    return 'Job is running'

Answer 1

p = subprocess.Popen([sys.executable, '/path/to/script.py'], 
                                    stdout=subprocess.PIPE, 
                                    stderr=subprocess.STDOUT)

That will start the subprocess in background. Your script will keep running normally.

Read the documentation here.

Answer 2

subprocess.Popen is indeed what you are looking for.

Answer 3

Although if you find that you want to start communicating a bunch of information between the subprocess and the parent, you may want to consider a thread, or RPC framework like Twisted.

But most likely those are too heavy for your application.

Answer 4

Running this through a message queue is definitely the way to go if you're thinking about long-term scaling. Send a message to the queue who's running constantly in the background, and write job handlers to deal with the different sorts of messages.

Since you're using Django, I think Beanstalkd is a pretty good fit. Here's a pretty nice tutorial on the subject. The first comment in that article also has some good tips.

Personally I've rolled with a custom in-memory queue server written in Erlang, with Python-bindings written in C. But redis looks like it might work out as a great contender for future queuing/messaging-needs. Hope this helps!