Unbuffered read from process using subprocess in Python

Question

I am trying to read from a process that produces long and time-consuming output. However, I want to catch it's output as and when it is produced. But using something like the following seems to be buffering the command's output, so I end up getting the output lines all at once:

p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, bufsize=0)
    for line in p.stdout:
        print line

I am trying this on MacOS 10.5

Answer 1

The file iterator is doing some internal buffering on its own. Try this:

line = p.stdout.readline()
while line:
    print line
    line = p.stdout.readline()

Answer 2

Usually, every program will do more buffering on its input and/or output channels than you appear to desire... unless it's fooled into believing said channel's actually a terminal!

For that "fooling in a good cause" purpose, use pexpect -- it works just fine on a Mac (life is harder on Windows, though there are solutions that might help even there - fortunately we don't need to dwell on those as you use a Mac instead).