programmatically executing and terminating a long-running batch process in python

Question

Hi guys,

I have been searching for a way to start and terminate a long-running "batch jobs" in python. Right now I'm using "os.system()" to launch a long-running batch job inside each child process. As you might have guessed, "os.system()" spawns a new process inside that child process (grandchild process?), so I cannot kill the batch job from the grand-parent process. To provide some visualization of what I have just described:

Main (grandparent) process, with PID = AAAA
          |
          |------> child process with PID = BBBB
                         |
                         |------> os.system("some long-running batch file)
                                  [grandchild process, with PID = CCCC]

So, my problem is I cannot kill the grandchild process from the grandparent...

My question is, is there a way to start a long-running batch job inside a child process, and being able to kill that batch job by just terminating the child process? What are the alternatives to os.system() that I can use so that I can kill the batch-job from the main process ?

Thanks !!

Answer 1

If you are on a Posix-compatible system (e.g., Linux or OS X) and no Python code has to be run after the child process, use os.execv. In general, avoid os.system and use the subprocess module instead.

Answer 2

If you want control over start and stop of child processes you have to use threading. In that case, look no further than Python's threading module.

Answer 3

subprocess module is the proper way to spawn and control processes in Python.

from the docs:

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several other, older modules and functions, such as:

os.system
os.spawn
os.popen
popen2
commands

so... if you are on Python 2.4+, subprocess is the replacement for os.system

for stopping processes, check out the terminate() and communicate() methods of Popen objects.