real time subprocess.Popen via stdout and PIPE

Question

I am trying to grab stdout from a subprocess,Popen call and although I am achieving this easily by doing:

cmd = subprocess.Popen('ls -l', shell=True, stdout=PIPE)
for line in cmd.stdout.readlines():
    print line

I would like to grab stdout in "real time". With the above method, PIPE is waiting to grab all the stdout and then it returns.

So for logging purposes, this doesn't meet my requirements (e.g. "see" what is going on while it happens).

Is there a way to get line by line, stdout while is running? Or is this a limitation of subprocess (having to wait until the PIPE closes).

EDIT If I switch readlines() for readline() I only get the last line of the stdout (not ideal):

In [75]: cmd = Popen('ls -l', shell=True, stdout=PIPE)
In [76]: for i in cmd.stdout.readline(): print i
....: 
t
o
t
a
l

1
0
4

Thanks!

Answer 1

The call to readlines is waiting for the process to exit. Replace this with a loop around cmd.stdout.readline() (note singular) and all should be well.

Answer 2

cmd = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE)
for line in cmd.stdout:
    print line.rstrip("\n")

Answer 3

Drop the readlines() which is coalescing the output. Also you'll need to enforce line buffering since most commands will interally buffer output to a pipe. For details see: http://www.pixelbeat.org/programming/stdio_buffering/

Answer 4

To get output "in real time", subprocess is unsuitable because it can't defeat the other process's buffering strategies. That's the reason I always recommend, whenever such "real time" output grabbing is desired (quite a frequent question on stack overflow!), to use instead pexpect (everywhere but Windows -- on Windows, wexpect).

Answer 5

Your interpreter is buffering. Add a call to sys.stdout.flush() after your print statement.