Python - capture Popen stdout AND display on console?

Question

Hi,

I want to capture stdout from a long-ish running process started via subprocess.Popen(...) so I'm using stdout=PIPE as an arg.

However, because it's a long running process I also want to send the output to the console (as if I hadn't piped it) to give the user of the script an idea that it's still working.

Is this at all possible?

Cheers.

Answer 1

Can you simply print it as you read it from the pipe?

Answer 2

The buffering your long-running sub-process is probably performing will make your console output jerky and very bad UX. I suggest you consider instead using pexpect (or, on Windows, wexpect) to defeat such buffering and get smooth, regular output from the sub-process. For example (on just about any unix-y system, after installing pexpect):

>>> import pexpect
>>> child = pexpect.spawn('/bin/bash -c "echo ba; sleep 1; echo bu"', logfile=sys.stdout); x=child.expect(pexpect.EOF); child.close()
ba
bu
>>> child.before
'ba\r\nbu\r\n'

The ba and bu will come with the proper timing (about a second between them). Note the output is not subject to normal terminal processing, so the carriage returns are left in there -- you'll need to post-process the string yourself (just a simple .replace!-) if you need \n as end-of-line markers (the lack of processing is important just in case the sub-process is writing binary data to its stdout -- this ensures all the data's left intact!-).

Answer 3

Alternatively, you can pipe your process into tee and capture only one of the streams. Something along the lines of "sh -c 'process interesting stuff'|tee /dev/stderr".

Of course, this only works on unix-like systems.

Answer 4

S. Lott's comment points to http://stackoverflow.com/questions/803265/getting-realtime-output-using-subprocess and http://stackoverflow.com/questions/1085071/real-time-intercepting-of-stdout-from-another-process-in-python

I'm curious that Alex's answer here is different from his answer 1085071. My simple little experiments with the answers in the two other referenced questions has given good results...

I went and looked at wexpect as per Alex's answer above, but I have to say reading the comments in the code I was not left a very good feeling about using it.

I guess the meta-question here is when will pexpect/wexpect be one of the Included Batteries?