Parsing commandline output progression containing carriage returns in real time with python

Question

I am able to transform carriage returns into new lines. The problem however is to get it running in nearly 'real time'. It will be quite stupid looking if progres bar only values are 0 and 100 :-)

This code returns output at once:

import subprocess

p = subprocess.Popen(['mplayer', '/home/user/sample.mkv'], stdout=subprocess.PIPE).communicate()[0]
for line in p.splitlines():
    if line.strip():
        print line

Answer 1

You need to do two things:

1. You must make sure that mplayer flushes the output for every line (should happen with progress output that gets printed into the same line).

2. You must read the output line by line. Instead of calling communicate(), you must close the p.stdin and then read p.stdout until EOF.

Answer 2

You are in for a world of pain with buffering in my experience. The reason being is that the standard C library will detect stdout isn't connected to a terminal and use more buffering. There is nothing you can do about that except hack the source of mplayer.

If you use python-pexpect though, it will start your subprocess using pseudo-ttys which the C library believes to be a terminal and it won't reset the buffering.

It is very easy to make subprocess deadlock when doing this sort of thing too which is another problem python-pexpect gets over.

Answer 3

pexpect anywhere but Windows, and wexpect on Windows, are always my recommendations when you need to "defeat buffering" and read a subprocess's output "in near real-time", as you put it. Since the subprocess you're running most likely buffers its output differently when it's outputting to a terminal vs. anything else (as that's the normal behavior of the C runtime libraries), you need to trick it into believing it IS outputting to a terminal rather than to your program, and that's what pexpect achieves (by building a pseudo-terminal via the lower-level pty module). I'm actually amazed that wexpect was able to do much the same on Windows, yet, while occasionally imperfect, it does also appear to work;-).

Answer 4

Ok, Thanks! I will keep on eye for pexpect. But I found out that there is cosplatform way to do it: PyQt4 and QProcess. While it is not certainly solution for every program, it certainly fits for Qt4 frontend application :)