How do I print outputs from calls to subprocess.Popen(...) in a loop?

Question

I wrote a script to run a command-line program with different input arguments and grab a certain line from the output. I have the following running in a loop:

p1 = subprocess.Popen(["program", args], stderr=subprocess.STDOUT, stdout=subprocess.PIPE, shell=False)
p2 = subprocess.Popen(["grep", phrase], stdin=p1.stdout, stdout=subprocess.PIPE, shell=False)
p1.wait()
p2.wait()
p = str(p2.stdout.readlines())
print 'p is ', p

One problem is that there is only output after the loop is finished running. I want to print something each time a process is finished. How can I do that?

Also, I want to have the option of displaying the output of p1. But I can't grab it with p1.stdout.readlines() without breaking p2. How can I do this?

I was thinking that I could just not make the call to grep, store the output of p1 and search for the phrase, but there's a lot of output, so this way seems pretty inefficient.

Any suggestions would be greatly appreciated. Thanks!

Answer 1

Try calling sys.stdout.flush() after each print statement.

Answer 2

Here's a quick hack that worked for me on Linux. It might work for you, depending on your requirements. It uses tee as a filter that, if you pass print_all to your script, will duplicate an extra copy to /dev/tty (hey, I said it was a hack):

#!/usr/bin/env python

import subprocess
import sys

phrase = "bar"
if len(sys.argv) > 1 and sys.argv[1] == 'print_all':
    tee_args = ['tee', '/dev/tty']
else:
    tee_args = ['tee']

p1 = subprocess.Popen(["./program"], stderr=subprocess.STDOUT, stdout=subprocess.PIPE, shell=False)
p2 = subprocess.Popen(tee_args, stdin=p1.stdout, stdout=subprocess.PIPE, shell=False)
p3 = subprocess.Popen(["grep", phrase], stdin=p2.stdout, stdout=subprocess.PIPE, shell=False)
p1.wait()
p2.wait()
p3.wait()
p = str(p3.stdout.readlines())
print 'p is ', p

With the following as contents for program:

#!/bin/sh

echo foo
echo bar
echo baz

Example output:

$ ./foo13.py
p is  ['bar\n']
$ ./foo13.py print_all
foo
bar
baz
p is  ['bar\n']