What is the fool proof way to convert some string (utf-8 or else) to a simple ASCII string in python

Question

Inside my python scrip, I get some string back from a function which I didn't write. The encoding of it varies. I need to convert it to ascii format. Is there some fool-proof way of doing this? I don't mind replacing the non-ascii chars with blanks or something else...

Answer 1

You say "the encoding of it varies". I guess that by "it" you mean a Python 2.x "string", which is really a sequence of bytes.

Answer part one: if you do not know the encoding of that encoded string, then no, there is no way at all to do anything meaningful with it*. If you do know the encoding, then step one is to convert your str into a unicode:

encoded_string = i_have_no_control()
the_encoding = 'utf-8' # for the sake of example
text = unicode(encoded_string, the_encoding)

Then you can re-encode your unicode object as ASCII, if you like.

ascii_garbage = text.encode('ascii', 'replace')

* There are heuristic methods for guessing encodings, but they are slow and unreliable. Here's one excellent attempt in Python.

Answer 2

If all you want to do is preserve ASCII-compatible characters and throw away the rest, then in most encodings that boils down to removing all characters that have the high bit set -- i.e., characters with value over 127. This works because nearly all character sets are extensions of 7-bit ASCII.

If it's a normal string (i.e., not unicode), you need to decode it in an arbitrary character set (such as iso-8859-1 because it accepts any byte values) and then encode in ascii, using the ignore or replace option for errors:

>>> orig = '1ä2äö3öü4ü'
>>> orig.decode('iso-8859-1').encode('ascii', 'ignore')
'1234'
>>> orig.decode('iso-8859-1').encode('ascii', 'replace')
'1??2????3????4??'

The decode step is necessary because you need a unicode string in order to use encode. If you already have a Unicode string, it's simpler:

>>> orig = u'1ä2äö3öü4ü'
>>> orig.encode('ascii', 'ignore')
'1234'
>>> orig.encode('ascii', 'replace')
'1??2????3????4??'

Answer 3

I'd try to normalize the string then encode it. What about :

import unicodedata
s = u"éèêàùçÇ"
print unicodedata.normalize('NFKD',s).encode('ascii','ignore')

This works only if you have unicode as input. Therefor, you must know what can of encoding the function ouputs and decode it. If you don't, there are encoding detection heuristics, but on short strings, there are not reliable.

Of course, you could have luck and the function outputs rely on various unknow encodings but using ascii as a code base, therefor they would allocate the same value for the bytes from 0 to 127 (like utf-8).

In that case, you can just get rid of the unwanted chars by filtering them using OrderedSets :

import string.printable # asccii chars
print "".join(OrderedSet(string.printable) & OrderedSet(s))

Or if you want blanks instead :

print("".join(((char if char in  string.printable else " ") for char in s )))

"translate" can help you to do the same.

The only way to know if your are this lucky is to try it out... Sometimes, a big fat lucky day is what any dev need :-)

Answer 4

If you want an ASCII string that unambiguously represents what you have got, without losing any information, the answer is simple:

Don't muck about with encode/decode, use the repr() function (Python 2.X) or the ascii() function (Python 3.x).