Using python to run other programs

Question

I have a command that works great on the command line. It has lots of arguments like cmd --thing foo --stuff bar -a b input output

I want to run this from python and block waiting for it to complete. As the script prints things to stdout and stderr I want it to be immediately shown to the user.

What is the right module for this?

I've tried:

---

import commands
output = commands.getoutput("cmd --thing foo --stuff bar -a b input output")
print output

this works great except the stdout isn't returned until the end.

---

import os
os.system("cmd --thing foo --stuff bar -a b input output")

this prints all the output when the cmd is actually finished.

---

import subprocess
subprocess.call(["cmd", "--thing foo", "--stuff bar", "-a b", "input", "output"])

this doesn't pass the parameters correctly somehow (I haven't been able to find the exact problem, but cmd is rejecting my input). If I put echo as the first parameter, it prints out the command which works perfectly when I paste it directly into the terminal.

---

import subprocess
subprocess.call("cmd --thing foo --stuff bar -a b input output")

exactly the same as above.

Answer 1

You have to quote each field separately, ie. split the options from their arguments.

import subprocess
output = subprocess.call(["cmd", "--thing", "foo", "--stuff", "bar", "-a", "b", "input", "output"])

otherwise you are effectively running cmd like this

$ cmd --thing\ foo --stuff\ bar -a\ b input output

To get the output into a pipe you need to call it slightly differently

import subprocess
output = subprocess.Popen(["cmd", "--thing", "foo", "--stuff", "bar", "-a", "b", "input", "output"],stdout=subprocess.PIPE)
output.stdout   #  <open file '<fdopen>', mode 'rb'>

Answer 2

Wouldn't commands.getstatusoutput() work? It'll return your status right away pretty sure.

Answer 3

A coworker just showed me this:

import os
import sys
for line in os.popen("cmd --thing foo --stuff bar -a b input output", "r"):
    print line
    sys.stdout.flush()

and it is working :)

Answer 4

If you don't need to process the output in your code, only to show it to the user as it happens (it's not clear from your Q, and it seems that way from your own self-answer), simplest is:

rc = subprocess.call(
    ["cmd", "--thing", "foo", "--stuff", "bar", 
     "-a", "b", "input", "output"])
print "Return code was", rc

i.e., just avoid any use of pipes -- let stdout and stderr just show on the terminal. That should avoid any problem with buffering. Once you put pipes in the picture, buffering generally is a problem if you want to show output as it happens (I'm surprised your self-answer doesn't have that problem;-).

For both showing and capturing, BTW, I always recomment pexpect (and wexpect on Windows) exactly to work around the buffering issue.