views:

391

answers:

4

I'm looking for a Python solution that will allow me to save the output of a command in a file without hiding it from the console.

FYI: I'm asking about tee (as the Unix command line utility) and not the function with the same name from Python intertools module.

Details

  • Python solution (not calling tee, it is not available under Windows)
  • I do not need to provide any input to stdin for called process
  • I have no control over the called program, all I know is that it will output something to stdout and stderr and return with an exit code.
  • to work when calling external programs (subprocess)
  • to work for both stderr and stdout
  • being able to differentiate between stdout and stderr because I may want to display only one of the to the console or I could try to output stderr using a different color - this means that stderr = subprocess.STDOUT will not work.
  • live output (progressive) - the process can run for a long time and I'm not able to wait for it to finish.
  • Python 3 compatible code (important)

References

Here are some incomplete solutions I found so far:

diagram

current code (2nd try)

#!/usr/bin/python
from __future__ import print_function

import sys, os, time, subprocess, io, threading
cmd = "python -E test_output.py"

from threading import Thread
class StreamThread ( Thread ):
    def __init__(self, buffer):
        Thread.__init__(self)
        self.buffer = buffer
    def run ( self ):
        while 1:
            line = self.buffer.readline()
            print(line,end="")
            sys.stdout.flush()
            if line == '':
                break

proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdoutThread = StreamThread(io.TextIOWrapper(proc.stdout))
stderrThread = StreamThread(io.TextIOWrapper(proc.stderr))
stdoutThread.start()
stderrThread.start()
proc.communicate()
stdoutThread.join()
stderrThread.join()

print("--done--")
#### test_output.py ####

#!/usr/bin/python
from __future__ import print_function
import sys, os, time

for i in range(0, 10):
    if i%2:
        print("stderr %s" % i, file=sys.stderr)
    else:
        print("stdout %s" % i, file=sys.stdout)
    time.sleep(0.1)
#### real output ####
stderr 1
stdout 0
stderr 3
stdout 2
stderr 5
stdout 4
stderr 7
stdout 6
stderr 9
stdout 8
--done--

Expected output was to have the lines ordered. Remark, modifying the Popen to use only one PIPE is not allowed because in the real life i will want to do different things with stderr and stdout.

Also even in the second case I was not able to obtain real-time like out, in fact all the results were received when the process finished. By default Popen should use no buffers (bufsize=0).

+2  A: 

This is a straightforward port of tee to Python.

import sys
sinks = sys.argv[1:]
sinks = [open(sink, "w") for sink in sinks]
sinks.append(sys.stderr)
while True:
  input = sys.stdin.read(1024)
  if input:
    for sink in sinks:
      sink.write(input)
  else:
    break

I'm running on Linux right now but this ought to work on most platforms.


Now for the subprocess part, I don't know how you want to 'wire' the subprocess's stdin, stdout and stderr to your stdin, stdout, stderr and file sinks, but I know you can do this:

import subprocess
callee = subprocess.Popen( ["python", "-i"],
                           stdin = subprocess.PIPE,
                           stdout = subprocess.PIPE,
                           stderr = subprocess.PIPE
                         )

Now you can access callee.stdin, callee.stdout and callee.stderr like normal files, enabling the above "solution" to work. If you want to get the callee.returncode, you'll need to make an extra call to callee.poll().

Be careful with writing to callee.stdin: if the process has exited when you do that, an error may be rised (on Linux, I get IOError: [Errno 32] Broken pipe).

badp
This is suboptimal in Linux, since Linux provides an ad-hoc [`tee(f_in, f_out, len, flags)`](http://linux.die.net/man/2/tee) API, but that's not the point right?
badp
I updated the question, the problem is that I was not able to find how to use subprocess in order to get the data from the two pipes gradually and not all at once at the end of the process.
Sorin Sbarnea
@Sorin, what if you replaced `read(1024)` with `read(1)`?
badp
I know that your code should work but there is a small requirement that does break the entire logic: I want to be able to distinguish between stdout and stderr and this means that I have to read from both of them but I do not know which will get new data. Please take a look at the example code.
Sorin Sbarnea
@Sorin, that means you'll have to either use two threads. One reads on `stdout`, one reads on `stderr`. If you are going to write both to the same file, you can acquire a lock on the sinks when you start reading and release it after writing a line terminator. :/
badp
Using threads for this does not sounds too appealing to me, maybe we'll find something else. It's strange that this is a common issue but nobody provided a complete solution for it.
Sorin Sbarnea
@badp I tried the threads a approach but it doesn't work. I updates the question to include the new example.
Sorin Sbarnea
@Sorin The output you have posted _is_ ordered. You had `line1 line3 line5 line7 line9` on stderr, `line0 line2 line4 line6 line8` on stdout. Sure, in that run the `stderr` thread happened to get output first, which meant you had `line1 line0 line3 line2 line5 line4...` instead of `line0 line1 line2 line3 line4 line5...` -- but you didn't get `line0 line3 line5 line1 line2...` or `line4 line2 line1 line0 line6...` or `line0 liline1 line3 linne2 line3e5...`. I'm afraid that for a program that has to accept arbitrary input this kind of nondeterminism is unaivoidable if not even necessary.
badp
+1  A: 

If you don't want to interact with the process you can use the subprocess module just fine.

Example:

tester.py

import os
import sys

for file in os.listdir('.'):
    print file

sys.stderr.write("Oh noes, a shrubbery!")
sys.stderr.flush()
sys.stderr.close()

testing.py

import subprocess

p = subprocess.Popen(['python', 'tester.py'], stdout=subprocess.PIPE,
                     stdin=subprocess.PIPE, stderr=subprocess.PIPE)

stdout, stderr = p.communicate()
print stdout, stderr

In your situation you can simply write stdout/stderr to a file first. You can send arguments to your process with communicate as well, though I wasn't able to figure out how to continually interact with the subprocess.

Wayne Werner
This doesn't show you error messages in STDERR in context of STDOUT, which can make debugging shell-scripts etc nearly impossible.
RobM
Meaning...? In this script anything delivered through STDERR is printed to the screen along with STDOUT. If you're referring to return codes, just use `p.poll()` to retrieve them.
Wayne Werner
+1  A: 

Try this :

import sys

class tee-function :

    def __init__(self, _var1, _var2) :

        self.var1 = _var1
        self.var2 = _var2

    def __del__(self) :

        if self.var1 != sys.stdout and self.var1 != sys.stderr :
            self.var1.close()
        if self.var2 != sys.stdout and self.var2 != sys.stderr :
            self.var2.close()

    def write(self, text) :

        self.var1.write(text)
        self.var2.write(text)

    def flush(self) :

        self.var1.flush()
        self.var2.flush()

stderrsav = sys.stderr

out = open(log, "w")

sys.stderr = tee-function(stderrsav, out)
Catalin Festila
This is exactly the approach I was about to suggest. Also worth adding some of the file data-descriptors, like `closed`.
RobM
Just tried it, `subprocess.Popen` calls `fileno()`, triggering an exception.
RobM
A: 

Finally I had to implement tee() command in Python myself.

You can get it from here http://github.com/ssbarnea/tendo/blob/master/tendo/tee.py

Currently it does allow you to do things like:

 tee("python --v") # works just like os.system()

 tee("python --v", "log.txt") # file names

 tee("python --v", file_handle)

 import logging
 tee("python --v", logging.info) # receive a method

The only current limitation is that it is not able to differentiate between stderr and stdout, meaning that it will merge both of them.

Sorin Sbarnea