What's the fastest way to strip and replace a document of high unicode characters using Python?

Question

I am looking to replace from a large document all high unicode characters, such as accented Es, left and right quotes, etc., with "normal" counterparts in the low range, such as a regular 'E', and straight quotes. I need to perform this on a very large document rather often. I see an example of this in what I think might be perl here: http://www.designmeme.com/mtplugins/lowdown.txt

Is there a fast way of doing this in Python without using s.replace(...).replace(...).replace(...)...? I've tried this on just a few characters to replace and the document stripping became really slow.

EDIT, my version of unutbu's code that doesn't seem to work:

# -*- coding: iso-8859-15 -*-
import unidecode
def ascii_map():
    data={}
    for num in range(256):
        h=num
        filename='x{num:02x}'.format(num=num)
        try:
            mod = __import__('unidecode.'+filename,
                             fromlist=True)
        except ImportError:
            pass
        else:
            for l,val in enumerate(mod.data):
                i=h<<8
                i+=l
                if i >= 0x80:
                    data[i]=unicode(val)
    return data

if __name__=='__main__':
    s = u'“fancy“fancy2'
    print(s.translate(ascii_map()))

Answer 1

.

# -*- encoding: utf-8 -*-
import unicodedata

def shoehorn_unicode_into_ascii(s):
    return unicodedata.normalize('NFKD', s).encode('ascii','ignore')

if __name__=='__main__':
    s = u"éèêàùçÇ"
    print(shoehorn_unicode_into_ascii(s))
    # eeeaucC

Note, as @Mark Tolonen kindly points out, the method above removes some characters like ß‘’“”. If the above code truncates characters that you wish translated, then you may have to use the string's translate method to manually fix these problems. Another option is to use unidecode (see J.F. Sebastian's answer).

When you have a large unicode string, using its translate method will be much much faster than using the replace method.

Edit: unidecode has a more complete mapping of unicode codepoints to ascii. However, unidecode.unidecode loops through the string character-by-character (in a Python loop), which is slower than using the translate method.

The following helper function uses unidecode's data files, and the translate method to attain better speed, especially for long strings.

In my tests on 1-6 MB text files, using ascii_map is about 4-6 times faster than unidecode.unidecode.

# -*- coding: utf-8 -*-
import unidecode
def ascii_map():
    data={}
    for num in range(256):
        h=num
        filename='x{num:02x}'.format(num=num)
        try:
            mod = __import__('unidecode.'+filename,
                             fromlist=True)
        except ImportError:
            pass
        else:
            for l,val in enumerate(mod.data):
                i=h<<8
                i+=l
                if i >= 0x80:
                    data[i]=unicode(val)
    return data

if __name__=='__main__':
    s = u"éèêàùçÇ"
    print(s.translate(ascii_map()))
    # eeeaucC

Edit2: Rhubarb, if # -*- encoding: utf-8 -*- is causing a SyntaxError, try # -*- encoding: cp1252 -*-. What encoding to declare depends on what encoding your text editor uses to save the file. Linux tends to use utf-8, and (it seems perhaps) Windows tends to cp1252.

Answer 2

There is no such thing as a "high ascii character". The ASCII character set is limited to ordinal in range(128).

That aside, this is a FAQ. Here's one answer. In general, you should familiarise yourself with str.translate() and unicode.translate() -- very handy for multiple substitutions of single bytes/characters. Beware of answers that mention only the unicodedata.normalize() gimmick; that's just one part of the solution.

Update: The currently-accepted answer blows away characters that don't have a decomposition, as pointed out by Mark Tolonen. There seems to be a lack of knowledge of what unicode.translate() is capable of. It CAN translate one character into multiple characters. Here is the output from help(unicode.translate):

S.translate(table) -> unicode

Return a copy of the string S, where all characters have been mapped through the given translation table, which must be a mapping of Unicode ordinals to Unicode ordinals, Unicode strings or None. Unmapped characters are left untouched. Characters mapped to None are deleted.

Here's an example:

>>> u"Gau\xdf".translate({0xdf: u"ss"})
u'Gauss'
>>>

Here's a table of fix-ups from the solution that I pointed to:

CHAR_REPLACEMENT = {
    # latin-1 characters that don't have a unicode decomposition
    0xc6: u"AE", # LATIN CAPITAL LETTER AE
    0xd0: u"D",  # LATIN CAPITAL LETTER ETH
    0xd8: u"OE", # LATIN CAPITAL LETTER O WITH STROKE
    0xde: u"Th", # LATIN CAPITAL LETTER THORN
    0xdf: u"ss", # LATIN SMALL LETTER SHARP S
    0xe6: u"ae", # LATIN SMALL LETTER AE
    0xf0: u"d",  # LATIN SMALL LETTER ETH
    0xf8: u"oe", # LATIN SMALL LETTER O WITH STROKE
    0xfe: u"th", # LATIN SMALL LETTER THORN
    }

This can be easily extended to cater for the fancy quotes and other non-latin-1 characters found in cp1252 and siblings.

Answer 3

If unicodedata.normalize() as suggested by ~unubtu doesn't do the trick, for example if you want more control over the mapping, you should look into
str.translate()
along with str.maketrans(), a utility to produce a map table, str.translate is both efficient and convenient for this type of translation.
In Python 2.x and for unicode strings one needs to use unicode.translate() rather than str.translate() and a trick similar to the one shown in the code snippet below, in lieu of maketrans(). (thanks to John Machin for pointing this out!)

These methods are also availble in in Python 3.x see for example the Python 3.1.2 documentation (for some reason I had made a mental note that this may have changed in Python 3.x). Of course under Python 3, all strings are unicode strings, but that's other issue.

#Python 3.1
>>> intab = 'àâçêèéïîôù'
>>> outtab = 'aaceeeiiou'
>>> tmap = str.maketrans(intab, outtab)
>>> s = "à la fête de l'été, où il fait bon danser, les Français font les drôles"
>>> s
"à la fête de l'été, où il fait bon danser, les Français font les drôles"
>>> s.translate(tmap)
"a la fete de l'ete, ou il fait bon danser, les Francais font les droles"
>>>


#Python 2.6
>>> intab = u'àâçêèéïîôù'
>>> outtab = u'aaceeeiiou'
>>> s = u"à la fête de l'été, où il fait bon danser, les Français font les drôles"
>>> #note the trick to replace maketrans() since for unicode strings the translation
>>> #     map expects integers (unicode ordinals) not characters.
>>> tmap = dict(zip(map(ord, intab), map(ord, outtab))) 
>>> s.translate(tmap)
u"a la fete de l'ete, ou il fait bon danser, les Francais font les droles"
>>>

Answer 4

Here's a solution that handles latin-1 characters (based on a 2003 usenet thread):

>>> accentstable = str.join("", map(chr, range(192))) + "AAAAAAACEEEEIIIIDNOOOOOxOUUUUYTsaaaaaaaceeeeiiiidnooooo/ouuuuyty"
>>> import string
>>> s = u"éèêàùçÇ"
>>> print string.translate(s.encode('latin1', 'ignore'), accentstable)
eeeaucC

Some of the mappings aren't perfect e.g. Thorn maps to T rather than Th, but it does a tolerable job.

Answer 5

I believe that unicodedata doesn't work for fancy quotes. You could use Unidecode in this case:

import unidecode
print unidecode.unidecode(u"ß‘’“”")
# -> ss''""