Case fold UTF-8 without knowing the language

views:

199

answers:

+3 Q:

Case fold UTF-8 without knowing the language

I'm trying to evaluate different strategies for case insensitive UTF-8 string comparison.

I've read some material from the Unicode consortium, experimented with ICU and tried to come up with various quality-of-implementation alternatives.

On multiple occasions I've seen texts differ between Simple Case Mapping and Full Case Mapping, and I wanted to make sure I understand the difference entirely.

As I read it, Simple Case Mapping is "context-free", i.e. doesn't need to know what language the payload is. This will give approximate results, due to the Turkic "I/ı/İ/i" debacle.

Full Case Mapping, on the other hand, needs to know the language of the payload to be able to perform the mapping. With that extra information, it can take special measures to cover cases where "Kim" as a Turkic string should become "KİM" in upper-case, but "Kim" as an English string, should become "KIM" in upper-case.

Have I got that right?

Are there other examples of "multi-faceted" code points that fold differently for different languages?

Thanks!

UPDATE: One of the sources mentioning simple case mapping as language independent is ICU's documentation. I interpreted that as Unicode truth, but maybe it's just a statement of the implementation?

+2 A:

Well ... The consonant combination "SS" would down-case to "ss" for most Western languages, but in German it might become the special letter "ß". That's just "might", there are quite involved usage rules to consider.

I think this doesn't directly affect collation order (any Germans are of course welcome to correct me) though, so maybe it's a moot point.

unwind 2009-11-25 08:52:45

Thanks! Did I understand the difference of Simple vs. Full mapping correctly?

Kim Gräsman 2009-11-25 09:07:16

Although uppercasing "ß" will give you "SS", I have seen no framework that does the oposite (lowercase("SS") to result in "ß").This is because sometimes is should be "ss" and the only way to decide is to have a full German dictionary. And sometimes even that is not enough (for instance both "weiss" and "weiß" are correct words). In fact, not even a human can lowercase "WEISS" without context (what it means).

Mihai Nita 2009-11-26 08:15:42

@Mihai -- thanks, that makes sense. I had the same thought, that uppering would be much easier than lowering.

Kim Gräsman 2009-11-26 10:42:41

+2 A:

No, a "full case mapping" is a casing where one codepoint needs to be replaced by more than one new codepoints. A simple case mapping is a single codepoint substitution.

If you want to implement this yourself then the Unicode CaseFolding.txt file is crucial to get this right. Note the status field code "T", specifically there to handle the Turkish I problem.

Hans Passant 2009-11-25 17:19:21

So they both need the language context, right? I use a third-party library (PCRE) that doesn't use CaseFolding.txt, but only the case information from UnicodeData.txt, and doesn't require language context (neither explicitly nor implicitly, as far as I can tell). I figured maybe that was a valid compromise in the Simple case.

Kim Gräsman 2009-11-26 06:47:21

Absolutely. As noted in the file, you'll need to know when to ignore the records with the "T" status code.

Hans Passant 2009-11-26 12:29:33

As far as I can see, the T status code appears in CaseFolding.txt, and not UnicodeData.txt. But are you really saying that _correct_ folding can only be done with knowledge of language context? I'm looking for a compromise that doesn't require the context, and isn't 100% perfect... But maybe that's the first step on the road to warmth?

Kim Gräsman 2009-11-26 15:10:04

ansaurus

tags:

views:

answers:

Case fold UTF-8 without knowing the language

related questions