Convert UTF-8 octets to unicode code points

Question

Hi,

I have a set of UTF-8 octets and I need to convert them back to unicode code points. How can I do this in python.

e.g. UTF-8 octet ['0xc5','0x81'] should be converted to 0x141 codepoint.

Answer 1

I'm assuming pre-3.x...

Put them in a str, and either call unicode with the string and 'utf-8':

>>> unicode('\xc5\x81', 'utf-8')
u'\u0141'

Or call .decode('utf-8') on the str:

>>> '\xc5\x81'.decode('utf-8')
u'\u0141'

If by "octet" you really mean a string in the form '0xc5' (rather than '\xc5') you can convert them like this:

>>> ''.join(chr(int(x,0)) for x in ['0xc5', '0x81'])
'\xc5\x81'

Answer 2

l = ['0xc5','0x81']
s = ''.join([chr(int(c, 16)) for c in l]).decode('utf8')
s
>>> u'\u0141'

Answer 3

>>> "".join((chr(int(x,16)) for x in ['0xc5','0x81'])).decode("utf8")
u'\u0141'

Answer 4

In lovely 3.x, where all strs are Unicode, and bytes are what strs used to be:

>>> s = str(bytes([0xc5, 0x81]), 'utf-8')
>>> s
'Ł'
>>> ord(s)
321
>>> hex(ord(s))
'0x141'

Which is what you asked for.