I'm looking for a Python solution that will allow me to save the output of a command in a file without hiding it from the console.
FYI: I'm asking about tee (as the Unix command line utility) and not the function with the same name from Python intertools module.
Details
- Python solution (not calling
tee
, it is not available under Windows) - I do not need to provide any input to stdin for called process
- I have no control over the called program, all I know is that it will output something to stdout and stderr and return with an exit code.
- to work when calling external programs (subprocess)
- to work for both
stderr
andstdout
- being able to differentiate between stdout and stderr because I may want to display only one of the to the console or I could try to output stderr using a different color - this means that
stderr = subprocess.STDOUT
will not work. - live output (progressive) - the process can run for a long time and I'm not able to wait for it to finish.
- Python 3 compatible code (important)
References
Here are some incomplete solutions I found so far:
- http://devlishgenius.blogspot.com/2008/10/logging-in-real-time-in-python.html (mkfifo works only on Unix)
- http://blog.kagesenshi.org/2008/02/teeing-python-subprocesspopen-output.html (doesn't work at all)
current code (2nd try)
#!/usr/bin/python from __future__ import print_function import sys, os, time, subprocess, io, threading cmd = "python -E test_output.py" from threading import Thread class StreamThread ( Thread ): def __init__(self, buffer): Thread.__init__(self) self.buffer = buffer def run ( self ): while 1: line = self.buffer.readline() print(line,end="") sys.stdout.flush() if line == '': break proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdoutThread = StreamThread(io.TextIOWrapper(proc.stdout)) stderrThread = StreamThread(io.TextIOWrapper(proc.stderr)) stdoutThread.start() stderrThread.start() proc.communicate() stdoutThread.join() stderrThread.join() print("--done--")
#### test_output.py #### #!/usr/bin/python from __future__ import print_function import sys, os, time for i in range(0, 10): if i%2: print("stderr %s" % i, file=sys.stderr) else: print("stdout %s" % i, file=sys.stdout) time.sleep(0.1)
#### real output #### stderr 1 stdout 0 stderr 3 stdout 2 stderr 5 stdout 4 stderr 7 stdout 6 stderr 9 stdout 8 --done--
Expected output was to have the lines ordered. Remark, modifying the Popen to use only one PIPE is not allowed because in the real life i will want to do different things with stderr and stdout.
Also even in the second case I was not able to obtain real-time like out, in fact all the results were received when the process finished. By default Popen should use no buffers (bufsize=0).