Setting the correct encoding when piping stdout in python

Question

When piping the output of a python program, the python interpreter gets confused about encoding and sets it to None. This means a program like this:

# -*- coding: utf-8 -*-
print "åäö"

will work fine when run normally, but fail with:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 0: ordinal not in range(128)

when used in a pipe sequence.

What is the best way to make this work when piping? Can I just tell it to use whatever encoding the shell/filesystem/whatever is using?

The suggestions I have seen thus far is to modify your site.py directly, or hardcoding the defaultencoding using this hack:

# -*- coding: utf-8 -*-
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
print "åäö"

Is there a better way to make piping work?

Answer 1

Your code works when run in an script because python encodes the output to whatever encoding your terminal application is using. If you are piping you must encode it yourself.

A rule of thumb is: Always use unicode internally. decode what you receive, encode what you send.

# -*- coding: utf-8 -*-
print u"åäö".encode('utf-8')

Another didactic example is a python program to convert between iso8859-1 and utf-8, making everything uppercase in between.

import sys
for line in sys.stdin:
    # decode what you receive:
    line = line.decode('iso8859-1')

    # work with unicode internally:
    line = line.upper()

    # encode what you send:
    line = line.encode('utf-8')
    sys.stdout.write(line)

Setting system default encoding is a bad idea because some modules and libraries you use can rely on the fact it is ascii. Don't do it.

Answer 2

First, regarding this solution:

# -*- coding: utf-8 -*-
print u"åäö".encode('utf-8')

It's not practical to explicitly print with a given encoding every time. That would be repetitive and error-prone.

A better solution is to change sys.stdout at the start of your program, to encode with a selected encoding. Here is one solution I found on Python: How is sys.stdout.encoding chosen?, in particular a comment by "toka":

import sys
import codecs
sys.stdout = codecs.getwriter('utf8')(sys.stdout)

Answer 3

You may want to try changing the environment variable "PYTHONIOENCODING" to "utf_8." I have written a page on my ordeal with this problem.