What's wrong with this string normilizer Python snippet?

Question

It seems that every time I think I mastered encoding, I find something new to puzzle me :-)

I'm trying to get rid of French accents from an UTF-8 string:

>>> import unicodedata

>>> s = u"éèêàùçÇ"

>>> print(unicodedata.normalize('NFKD', s).encode('ascii','ignore'))

I expected eeeaucC as an output and got instead AA AaA A1AA using Python 2.6.4 in Ubuntu 9.10 and iPython 0.10, all the stuff set to unicode.

Answer 1

Afters further tests, it works if you use Python 3 or Python 2.6 interpreters instead of iPython.

Maybe a wrong user setting or a bug.

Answer 2

python works as it should:

$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:43:55) 
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s = u"éèêàùçÇ"
>>> s
u'\xe9\xe8\xea\xe0\xf9\xe7\xc7'
>>> ord(s[0])
233

There is some bug in ipython:

$ ipython
Python 2.6.4 (r264:75706, Dec  7 2009, 18:43:55) 
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: s = u"éèêàùçÇ"

In [2]: ord(s[0])
Out[2]: 195

In [3]: s
Out[3]: u'\xc3\xa9\xc3\xa8\xc3\xaa\xc3\xa0\xc3\xb9\xc3\xa7\xc3\x87'

If you read it from file then ipython works:

$ ipython
...
In [1]: import codecs

In [2]: s = codecs.open('s.txt', 'r', 'utf-8').read()

In [3]: s
Out[3]: u'\xe9\xe8\xea\xe0\xf9\xe7\xc7'

In [4]: ord(s[0])
Out[4]: 233