ansaurus

Question

Answer 1

+2 A:

If html is not already a unicode object but a normal string, it is converted to unicode when it is concatenated with the converted version of rs[0]. If html now contains special characters you can get a unicode error.

So the other contents of html also need to be correctly decoded to unicode. If the special characters come from string literals, you could use unicode literals (like u"abcä") instead.

sth 2010-05-25 11:39:56

So what is the encoding of html shall I set with, so the html content displayed properly in mail?

e.b.white 2010-05-25 12:44:14

@zhangzhong: You should use the encoding the HTML is encoded in. Only you can know what encoding that is, but if its the same as the other data its probably 'big5'.

sth 2010-05-25 21:51:29

Answer 2

+1 A:

Your call to unicode() is working correctly. It is the concatenation, which is adding a unicode object to a byte string, that is causing trouble. If you change the first line to u'''<tr><td>''', (or u'<tr><td>') it should work fine.

Edit: This means your error lies in the data that is already in html by the time python reaches this snippet:

>>> '\x9f<tr><td>' + unicode('\xc3\x60', 'big5')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x9f in position 0: ordinal not in range(128)
>>> u'\x9f<tr><td>' + unicode('\xc3\x60', 'big5')
u'\x9f<tr><td>\u56a5'
>>>

jcdyer 2010-05-25 15:13:01

You should always deal with unicode objects when possible. Decode input to unicode as soon as you can. Encode output to a specific encoding (utf-8, for instance) as late as you can. Use `codecs.open()` instead of the built-in `open()`, whenever possible.

jcdyer 2010-05-25 15:14:46

ansaurus

tags:

views:

answers:

The confusion on python encoding

related questions