How to redirect stderr in Python?

Question

I would like to log all the output of a Python script. I tried:

import sys

log = []

class writer(object):
    def write(self, data):
        log.append(data)

sys.stdout = writer()
sys.stderr = writer()

Now, if I "print 'something' " it gets logged. But if I make for instance some syntax error, say "print 'something# ", it wont get logged - it will go into the console instead.

How do I capture also the errors from Python interpreter?

I saw a possible solution here:

http://www.velocityreviews.com/forums/showpost.php?p=1868822&postcount=3

but the second example logs into /dev/null - this is not what I want. I would like to log it into a list like my example above or StringIO or such...

Also, preferably I don't want to create a subprocess (and read its stdout and stderr in separate thread).

Answer 1

You can't do anything in Python code that can capture errors during the compilation of that same code. How could it? If the compiler can't finish compiling the code, it won't run the code, so your redirection hasn't even taken effect yet.

That's where your (undesired) subprocess comes in. You can write Python code that redirects the stdout, then invokes the Python interpreter to compile some other piece of code.

Answer 2

I can't think of an easy way. The python process's standard error is living on a lower level than a python file object (C vs. python).

You could wrap the python script in a second python script and use subprocess.Popen. It's also possible you could pull some magic like this in a single script:

import os
import subprocess
import sys

cat = subprocess.Popen("/bin/cat", stdin=subprocess.PIPE, stdout=subprocess.PIPE)
os.close(sys.stderr.fileno())
os.dup2(cat.stdin.fileno(), sys.stderr.fileno())

And then use select.poll() to check cat.stdout regularly to find output.

Yes, that seems to work.

The problem I foresee is that most of the time, something printed to stderr by python indicates it's about to exit. The more usual way to handle this would be via exceptions.

---------Edit

Somehow I missed the os.pipe() function.

import os, sys
r, w = os.pipe()
os.close(sys.stderr.fileno())
os.dup2(w, sys.stderr.fileno())

Then read from r

Answer 3

I have a piece of software I wrote for work that captures stderr to a file like so:

import sys
sys.stderr = open('C:\\err.txt', 'w')

so it's definitely possible.

I believe your problem is that you are creating two instances of writer.

Maybe something more like:

import sys

class writer(object):
    log = []

    def write(self, data):
        self.log.append(data)

logger = writer()
sys.stdout = logger
sys.stderr = logger