Python UTF-8 comparison

Question

a = {"a":"çö"}
b = "çö"
a['a']
>>> '\xc3\xa7\xc3\xb6'

b.decode('utf-8') == a['a']
>>> False

What is going in there?

edit= I'm sorry, it was my mistake. It is still False. I'm using Python 2.6 on Ubuntu 10.04.

Answer 1

b is a string, a is a dict

You want (I believe):

b == a['a']

Answer 2

Try b == a['a']

Answer 3

You are comparing a string to a dict.

>>> a = {"a":"çö"}
>>> b = "çö"
>>> a == b
False
>>> a['a'] == b
True

If you compare the string (b) to the member of a (a['a']), then you get the desired result.

Answer 4

Possible solutions

Either write like this:

a = {"a": u"çö"}
b = "çö"
b.decode('utf-8') == a['a']

Or like this (you may also skip the .decode('utf-8') on both sides):

a = {"a": "çö"}
b = "çö"
b.decode('utf-8') == a['a'].decode('utf-8')

Or like this (my recommendation):

a = {"a": u"çö"}
b = u"çö"
b == a['a']

Explanation

Updated based on Tim's comment. In your original code, b.decode('utf-8') == u'çö' and a['a'] == 'çö', so you're actually making the following comparison:

u'çö' == 'çö'

One of the objects is of type unicode, the other is of type str, so in order to execute the comparison, the str is converted to unicode and then the two unicode objects are compared. It works fine in the case of purely ASCII strings, for example: u'a' == 'a', since unicode('a') == u'a'.

However, it fails in case of u'çö' == 'çö', since unicode('çö') returns the following error: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128), and therefore the whole comparison returns False and issues the following warning: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal.

Answer 5

UTF-8 is an encoding used to record Unicode text in files. However, in Python you are working with objects that have a fixed way to represent Unicode text, and that way is not UTF-8.

You can still compare Unicode strings in Python, but this is unrelated to UTF-8, except that if you want to put constants into these Unicode strings, then you will need to encode the text of the file containing your source code, in UTF-8. As soon as the assignment operator is executed, the string is no longer UTF-8, but is now the Python internal representation.

By the way, if you are doing comparisons with Unicode, you probably will want to use the unicodedata module and normalize the strings before comparisons are done.

Answer 6

Make sure your code is in UTF-8 (NOT Latin-1) and/or use a coding line as so:

#! /usr/bin/python
# -*- coding: utf-8 -*-
a = {"a": u"çö"}
b = "çö"
assert b == a['a']
assert b.decode('utf-8') == a['a'].decode('utf-8')

If you're using unicode across the board, you can import unicode_literals from the future and cut back on encoding heartaches:

#! /usr/bin/python
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
a = {"a": u"çö"}
b = "çö"
assert b == a['a']
assert b == a['a']
assert b.encode('utf-8') != a['a']
assert b.encode('utf-8') == a['a'].encode('utf-8')

If a file uses unicode_literals, all "strings" are now u"unicode" objects (per the coding of the file) if they're not b"prepended" with a b (to emulate the string/bytes split in Python 3.X).

Answer 7

NullUserException is right that this should be correct:

b == a['a']

You're still getting "False" because you're decoding one side as utf-8 (creating a Unicode string) while the other side remains a utf-8 encoded byte string.