Python subprocess.Popen communicate through a pipeline

Question

I want to be able to use Popen.communicate and have the stdout logged to a file (in addition to being returned from communicate().

This does what I want - but is it really a good idea?

cat_task = subprocess.Popen(["cat"],  stdout=subprocess.PIPE, stdin=subprocess.PIPE)
tee_task = subprocess.Popen(["tee", "-a", "/tmp/logcmd"], stdin=cat_task.stdout, 
    stdout = subprocess.PIPE, close_fds=True)
cat_task.stdout = tee_task.stdout #since cat's stdout is consumed by tee, read from tee.
cat_task.communicate("hello there")
('hello there', None)

Any issues with this, looking at communicate's impl it looks good. But is there a nicer way?

Answer 1

Depending on your definition of "nicer", I would say that the following is probably nicer in the sense that it avoids having an additional tee process:

import subprocess

def logcommunicate(self, s):
    std = self.oldcommunicate(s)
    self.logfilehandle.write(std[0])
    return std

subprocess.Popen.oldcommunicate = subprocess.Popen.communicate
subprocess.Popen.communicate = logcommunicate
logfh = open("/tmp/communicate.log", "a")

proc = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
proc.logfilehandle = logfh

result = proc.communicate("hello there\n")
print result

In a nutshell, it provides a wrapper for communicate() that writes stdout to a file handle of your choice, and then returns the original tuple for you to use. I've omitted exception handling; you should probably add that if the program is more critical. Also, if you expect to create several Popen objects and want them all to log to the same file, you should probably arrange for logcommunicate() to be thread-safe (synchronised per file handle). You can easily extend this solution to write to separate files for stdout and stderr.

Note that if you expect to pass a lot of data back and forth, then communicate() might not be the best option since it buffers everything in memory.