How can I check if a Python unicode string contains non-Western letters?

Question

I have a Python Unicode string. I want to make sure it only contains letters from the Roman alphabet (A through Z), as well as letters commonly found in European alphabets, such as ß, ü, ø, é, à, and î. It should not contain characters from other alphabets (Chinese, Japanese, Korean, Arabic, Cyrillic, Hebrew, etc.). What's the best way to go about doing this?

Currently I am using this bit of code, but I don't know if it's the best way:

def only_roman_chars(s):
    try:
        s.encode("iso-8859-1")
        return True
    except UnicodeDecodeError:
        return False

(I am using Python 2.5. I am also doing this in Django, so if the Django framework happens to have a way to handle such strings, I can use that functionality -- I haven't come across anything like that, however.)

Answer 1

check the code in django.template.defaultfilters.slugify

import unicodedata
value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore')

is what you are looking for, you can then compare the resulting string with the original

Answer 2

You can use unicodedata to check character category

import unicodedata
unicodedata.category(u'a') # returns 'Ll'
unicodedata.category(u'א') # returns 'Lo'

Answer 3

For what you say you want to do, your approach is about right. If you are running on Windows, I'd suggest using cp1252 instead of iso-8859-1. You might also allow cp1250 as well -- this would pick up eastern European countries like Poland, Czech Republic, Slovakia, Romania, Slovenia, Hungary, Croatia, etc where the alphabet is Latin-based. Other cp125x would include Turkish and Maltese ...

You may also like to consider transcription from Cyrillic to Latin; as far as I know there are several systems, one of which may be endorsed by the UPU (Universal Postal Union).

I'm a little intrigued by your comment "Our shipping department doesn't want to have to fill out labels with, e.g., Chinese addresses" ... three questions: (1) do you mean "addresses in country X" or "addresses written in X-ese characters" (2) wouldn't it be better for your system to print the labels? (3) how does the order get shipped if it fails your test?

Answer 4

Checking for ISO-8559-1 would miss reasonable Western characters like 'œ' and '€'. The solution depends on how you define "Western", and how you want to handle non-letters. Here's one approach:

import unicodedata

def is_permitted_char(char):
    cat = unicodedata.category(char)[0]
    if cat == 'L': # Letter
        return 'LATIN' in unicodedata.name(char, '').split()
    elif cat == 'N': # Number
        # Only DIGIT ZERO - DIGIT NINE are allowed
        return '0' <= char <= '9'
    elif cat in ('S', 'P', 'Z'): # Symbol, Punctuation, or Space
        return True
    else:
        return False

def is_valid(text):
    return all(is_permitted_char(c) for c in text)

Answer 5

Maybe this will do if you're a django user?

from django.template.defaultfilters import slugify 

def justroman(s):
  return len(slugify(s)) == len(s)

Answer 6

import unicodedata as ud

latin_letters= {}

def is_latin(uchr):
    try: return latin_letters[uchr]
    except KeyError:
         return latin_letters.setdefault(uchr, 'LATIN' in ud.name(uchr))

def only_roman_chars(unistr):
    return all(is_latin(uchr)
           for uchr in unistr
           if uchr.isalpha()) # isalpha suggested by John Machin

>>> only_roman_chars(u"ελληνικά means greek")
False
>>> only_roman_chars(u"frappé")
True
>>> only_roman_chars(u"hôtel lœwe")
True
>>> only_roman_chars(u"123 ångstrom ð áß")
True
>>> only_roman_chars(u"russian: гага")
False