python subprocess with shell=True: redirections and platform-independent subprocess killing

Question

Hi,

I'm having a hard time getting what I want out of the python subprocess module (which supposed to an unified/platform-independent abstraction, afaik, but don't get me started on that :)).

So the simple thing what I'm after is the following. I want to

• Launch an external (stdio) application (possibly with subprocess), where I use shell-style redirections (like './myapp >stdout_log >stderr_log')
  • Basically i want to execute a shell command-line, so I have to specify shell=True for subprocess.Popen() (or else redirections in the command-line won't work)
• I want to launch this command line in an async fashion (so it runs as an independent sub-process, but my python process won't wait for it's completion)
• (My parent python process would look at the child process's logs from time to time to extract information, but this is irrelevant to the question)
• If my parent python process decides, it should be able to terminate this child process.

Now, my main problems are that

• I'm basically forced to use shell=True, to get redirections to work
• Processing the child's stdout/stderr in the parent python process is not an option, since I couldn't find functionality for doing it in a non-waiting way (and the parent python process must do other things while the child is running)
• If I use shell=True then subprocess.kill() will only terminate the shell but not the child process
• I'd need a reliable child process termination method that works on any platform (but at least linux and windows)

I hope I was specific enough. Thanks for any tips/hints in advance -- I just spent a whole day with subprocess, and IMHO it's a pain far from either platform-independent or simple :( (but perhaps it's just me)

UPDATE (2010-10-13):

If you launch a sub-process (even with shell=False), then the subprocess.Popen.kill() function will only kill that sub-process (so if there are any "grandchild" processes, they won't be terminated.)

I read about using the preexec_fn parameter to set the sid on all child processes, but it's unix-only: http://stackoverflow.com/questions/3876886/timeout-a-subprocess

Answer 1

Last time I was in a similar situation, I found out the easiest (and practically the only) solution was to kick off a thread which takes care of your child process. You can take different routes with this method, be it to parse the piping of the shell-style command and perform those in python code (which you said wasn't an option due to the blocking), which would at the same time fix your killing problem. Basically, thread encapsulation seems like the way to go.

Sadly my experience with subprocess is all on the Windows platform, which has tons of its own little quirks. subprocess has a lot of flaws all-around it seems, although it must do an okay job given the existence of the popen, popen2 and so forth modules that it is supposed to replace.

Answer 2

Going through your problems one at a time:

I'm basically forced to use shell=True, to get redirections to work

You can't just use the stdout and stderr parameters?

out_log = open("stdout_log", "w")
err_log = open("stderr_log", "w")
subproc = subprocess.popen(..., stdout=out_log, stderr=err_log, ...)

Processing the child's stdout/stderr in the parent python process is not an option, since I couldn't find functionality for doing it in a non-waiting way (and the parent python process must do other things while the child is running)

This is because of Windows. On Unix-type OSes, you just use the select module. Windows can only select on sockets, not files.

If I use shell=True then subprocess.kill() will only terminate the shell but not the child process

Because when shell=True the shell is the child process, and the commands are its children.

I'd need a reliable child process termination method that works on any platform (but at least linux and windows)

Does a reliable child process termination method even exist for Windows? Last I heard, even Task Manager's End Task wasn't 100% reliable. And in Linux, you might not be able to kill a process that has a file open via a crashed driver.

Like Stigma said, for Windows support you will need to use threads as subprocess proxies. Also, you should try to run with shell=False, and if you can't, please elaborate as to why not.

Answer 3

Answering some of the issues:

• If you want to deal with standard output and input from the command that is run as part of subprocess, you have to use "shell = true"

• Specific solution may depend on what you are trying to do. How ever, for processing output from a command that will run for a long time, there are some useful solutions

  • you could use poll, look at SO discussion: http://stackoverflow.com/questions/803265
  • Look at the following for asynchronous interaction with subprocess: http://code.activestate.com/recipes/576957-asynchronous-subprocess-using-asyncore/
  • The following augmentation of subprocess can also allow for async operations : http://code.google.com/p/subprocdev/
  • http://pypi.python.org/pypi/drainers/0.0.3
  • Other useful discussion on SO : http://stackoverflow.com/questions/3575554/python-subprocess-with-timeout-and-large-output-64k
  • As suggested, you could combine with the threads

• Popen instance provides a pid attribute with the process id of the child process as shell=True. You should be able to collect the command pid and use it with os.kill()

-