exceptions with python unicode encode/decode functions (why doesn't errors=ignore actually ignore them??)

Question

Does anyone know why the string conversion functions throw exceptions when errors="ignore" is passed? How can I convert from regular Python string objects to unicode without errors being thrown? Thanks very much!

python -c "import codecs; codecs.open('tmp', 'wb', encoding='utf8', errors='ignore').write('кошка')"

returns
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/codecs.py", line 686, in write
return self.writer.write(data)
File "/usr/lib/python2.6/codecs.py", line 351, in write
data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)

EDIT -- thanks for the responses, but does anyone know how to convert the literal above, not using the "u" prefix? The reason being is that you could, of course, be dealing with something that wasn't a constant :)

Answer 1

The write method (in Python 2) takes a unicode object, and you're passing it a str -- so the encode call in codecs.py line 351 is first trying to build a unicode object (with the default codec, 'ascii'). Fix is easy: change the write call to

write(u'кошка')

The u prefix tells Python you're using a Unicode object, and it should be fine.

Answer 2

problem is here ===>>>> write('кошка')

You are writing a str object, the recipient is expecting a unicode object, so it tries to convert it to unicode using the default encoding (ascii), which of course (?) produces the well-known (?) UnicodeDecodeError: 'ascii' codec can't decode byte 0xXX in position 0: ordinal not in range(128)

The whole point of using the codecs module like that is to get it to convert your unicode objects to utf8-encoded on the fly -- so feed it unicode

Update How to convert the literal or non-literal:

unicode_object = literal_or_whatever.decode("UNKNOWN_ENCODING")

Do you know how your literal is encoded? Would you like to tell us what you are trying to accomplish? A one liner with python -c isn't much help ;-)

Answer 3

a non-solution (from question author) I just found out: use python3

python3 -c "import codecs; codecs.open('tmp', 'wb', encoding='utf8', errors='ignore').write('кошка')"

Answer 4

In Python 2.x use write('кошка'.decode('utf-8') instead of write('кошка').

You can use other encoding too instead of 'utf-8'.

Hopefully it will not throw any error ...