How do I reverse Unicode decomposition using Python?

Question

Using Python 2.5, I have some text in stored in a unicode object:

Dinis e Isabel, uma difı´cil relac¸a˜o conjugal e polı´tica

This appears to be decomposed Unicode. Is there a generic way in Python to reverse the decomposition, so I end up with:

Dinis e Isabel, uma difícil relação conjugal e política

Answer 1

I can't really give you a definitive answer to your question because I never tried that. But there is a unicodedata module in the standard library. It has two functions decomposition() and normalize() that might help you here.

Edit: Make sure that it really is decomposed unicode. Sometimes there are weird ways to write characters that can't be directly expressed in an encoding. Like "a which is meant to be mentally parsed by a human or some specialized program as ä.

Answer 2

I think you are looking for this:

>>> import unicodedata    
>>> print unicodedata.normalize("NFC",u"c\u0327")
ç

Answer 3

Unfortunately it seems I actually have (for example) \u00B8 (cedilla) instead of \u0327 (combining cedilla) in my text.

Eurgh, nasty! You can still do it automatically, though the process wouldn't be entirely lossless as it involves a compatibility decomposition (NFKD).

Normalise U+00B8 to NFKD and you'll get a space followed by the U+0327. You could then scan through the string looking for any case of space-followed-by-combining-character, and remove the space. Finally recompose to NFC to put the combining characters onto the previous character instead.

s= unicodedata.normalize('NFKD', s)
s= ''.join(c for i, c in enumerate(s) if c!=' ' or unicodedata.combining(s[i+1])==0)
s= unicodedata.normalize('NFC', s)