How do I parse only foreign characters from the text in an HTML file with regular expressions

Question

I'm trying to parse HTML and automatically change the font of any foreign characters, and I'm having some issues. There are a few different hackish ways I'm trying to accomplish this, but none work really well, and I'm wondering if anyone has any ideas. Is there any easy way with python to match all the foreign characters (specifically, Japanese Kanji/Hirigana/Katakana) with regular expressions? What I've been using is the complement of a set of non-foreign characters ([^A-Za-z0-9 <>'"=]), but this isn't working well, and I'm worried it will match things enclosed in <...>, which I don't want to do.

Answer 1

I wouldn't use just regular expressions for this. Down that path lies an angry Tony the Pony.

I'd use an HTML parser in conjuction with regular expressions, though. That way you can distinguish the markup from the non-markup.

Answer 2

Use BeautifulSoup to get the content that you need, then use a variation on this code to match your characters.

import re

kataLetters = range(0x30A0, 0x30FF)
hiraLetters = range(0x3040, 0x309F)
kataPunctuation = range(0x31F0,0x31FF)

myLetters = kataLetters+kataPunctuation+hiraLetters

myLetters = u''.join([unichr(aLetter) for aLetter in myLetters])


myRe = re.compile('['+myLetters+']+', re.UNICODE)

Use the code charts here to get the ranges for your characters.