Replace numeric character references in XML document using Python

Answer 1

Eurgh. You've got surrogates (UTF-16 code units in the range D800-DFFF), which some fool has incorrectly encoded individually instead of using a pair of code units for a single character. It would be ideal to replace this mess with what it should look like:

<tag>&#66352;&#66365;&#66355;&#66352;&#66367;&#66370;&#66352;</tag>

Or, just as valid, in literal characters (if you've got a font that can display the Gothic alphabet):

<tag></tag>

Usually, it would be best to do replacement operations like this on parsed text nodes, to avoid messing up non-character-reference sequences in other places like comments or PIs. However of course that's not possible in this case since this isn't really XML at all. You could try to fix it up with a crude regex, though it would be better to find out where the invalid input is coming from and kick the person responsible until they fix it.

>>> def lenient_deccharref(m):
...    return unichr(int(m.group(1)))
...
>>> tag= '<tag>&#55296;&#57136;&#55296;&#57149;&#55296;&#57139;&#55296;&#57136;&#55296;&#57151;&#55296;&#57154;&#55296;&#57136;</tag>'
>>> re.sub('&#(\d+);', lenient_deccharref, tag).encode('utf-8')
'<tag>\xf0\x90\x8c\xb0\xf0\x90\x8c\xbd\xf0\x90\x8c\xb3\xf0\x90\x8c\xb0\xf0\x90\x8c\xbf\xf0\x90\x8d\x82\xf0\x90\x8c\xb0</tag>'

This is the correct UTF-8 encoding of . The utf-8 codec allows you to encode a sequence of surrogates to correct UTF-8 even on a wide-Unicode platform where the surrogates should not have appeared in the string in the first place.

>>> _.decode('utf-8')
u'<tag>\U00010330\U0001033d\U00010333\U00010330\U0001033f\U00010342\U00010330</tag>'