How to write script output to file and command-line?

Question

I have a long-running Python script that I run from the command-line. The script writes progress messages and results to the standard output. I want to capture everything the script write to the standard output in a file, but also see it on the command line. Alternatively, I want the output to go to the file immediately, so I can use tail to view the progress. I have tried this:

python MyLongRunngingScript.py | tee log.txt

But it does not produce any output (just running the script produces output as expected). Can anyone propose a simple solution? I am using Mac OS X 10.6.4.

Edit I am using print for output in my script.

Answer 1

You could try doing sys.stdout.flush() occasionally in your script, and running with tee again. When stdout is redirected through to tee, it might get buffered for longer than if it's going straight to a terminal.

Answer 2

The fact that you don't see anything is probably related to the fact that buffering is occurring. So you only get output every 4 Ko of text or so.

instead, try something like this :

class OutputSplitter(object):
    def __init__(self, real_output, *open_files):
        self.__stdout = real_output
        self.__fds = open_files
        self.encoding = real_output.encoding
    def write(self, string):
        self.__stdout.write(string) # don't catch exception on that one.
        self.__stdout.flush()
        for fd in self.__fds:
            try:
                fd.write(string)
                fd.flush()
            except IOError:
                pass # do what you want here. 
    def flush(self):
        pass # already flushed

Then decorate sys.stdout with that class with some code like that :

stdout_saved = sys.stdout
logfile = open("log.txt","a") # check exception on that one.
sys.stdout = OutputSplitter(stdout_saved, logfile)

That way, every output (print included) is flushed to the standard output and to the specified file. Might require tweaking because i haven't tested that implementation.

Of course, expect to see a (small most of the time) performance penalty when printing messages.

Answer 3

You are on the right path but the problem is python buffering the output.

Fortunately there is a way to tell it not to buffer output:

python -u MyLongRunngingScript.py | tee log.txt