Python subprocess.Popen hangs in 'for l in p.stdout' until p terminates, why?

Question

I have that code:

#!/usr/bin/python -u

localport = 9876

import sys, re, os
from subprocess import *

tun = Popen(["./newtunnel", "22", str(localport)], stdout=PIPE, stderr=STDOUT)

print "** Started tunnel, waiting to be ready ..."
for l in tun.stdout:
        sys.stdout.write(l)
        if re.search("Waiting for connection", l):
                print "** Ready for SSH !"
                break

The "./newtunnel" will not exit, it will constantly output more and more data to stdout. However, that code will not give any output and just keeps waiting in the tun.stdout.

When I kill the newtunnel process externally, it flushes all the data to tun.stdout. So it seems that I can't get any data from the tun.stdout while it is still running.

Why is that? How can I get the information?

Note that the default bufsize for Popen is 0 (unbuffered). I can also specify bufsize=0 but that doesn't change anything.

Answer 1

Ok, it seems that it is a bug in Python: http://bugs.python.org/issue3907

If I replace the line

for l in tun.stdout:

by

while True:
    l = tun.stdout.readline()

then it works exactly the way I want.