Enumerating a string by grapheme instead of character

Question

Strings are usually enumerated by character. But, particuarly when working with Unicode and non-English languages, sometimes I need to enumerate a string by grapheme. That is, combining marks and diacritics should be kept with the base character they modify. What is the best way to do this in .Net?

Use case: Count the distinct phonetic sounds in a series of IPA words.

1. Simplified definition: There is a one-to-one relationship between a grapheme and a sound.
2. Realistic definition: Special "letter-like" characters should also be included with the base character (ex. pʰ), and some sounds may be represented by two symbols joined by a tie bar (k͡p).

Answer 1

Simplified scenario

The TextElementEnumerator is very useful and efficient:

private static List<SoundCount> CountSounds(IEnumerable<string> words)
{
    Dictionary<string, SoundCount> soundCounts = new Dictionary<string, SoundCount>();

    foreach (var word in words)
    {
        TextElementEnumerator graphemeEnumerator = StringInfo.GetTextElementEnumerator(word);
        while (graphemeEnumerator.MoveNext())
        {
            string grapheme = graphemeEnumerator.GetTextElement();

            SoundCount count;
            if (!soundCounts.TryGetValue(grapheme, out count))
            {
                count = new SoundCount() { Sound = grapheme };
                soundCounts.Add(grapheme, count);
            }
            count.Count++;
        }
    }

    return new List<SoundCount>(soundCounts.Values);
}

You can also do this using a regular expression: (From the documentation, the TextElementEnumerator handles a few cases that the expression below does not, particularly supplementary characters, but those are pretty rare, and in any case not needed for my application.)

private static List<SoundCount> CountSoundsRegex(IEnumerable<string> words)
{
    var soundCounts = new Dictionary<string, SoundCount>();
    var graphemeExpression = new Regex(@"\P{M}\p{M}*");

    foreach (var word in words)
    {
        Match graphemeMatch = graphemeExpression.Match(word);
        while (graphemeMatch.Success)
        {
            string grapheme = graphemeMatch.Value;

            SoundCount count;
            if (!soundCounts.TryGetValue(grapheme, out count))
            {
                count = new SoundCount() { Sound = grapheme };
                soundCounts.Add(grapheme, count);
            }
            count.Count++;

            graphemeMatch = graphemeMatch.NextMatch();
        }
    }

    return new List<SoundCount>(soundCounts.Values);
}

Performance: In my testing, I found that the TextElementEnumerator was about 4 times as fast as the regular expression.

Realistic scenario

Unfortunately, there is no way to "tweak" how the TextElementEnumerator enumerates, so that class will be of no use in the realistic scenario.

One solution is to tweak our regular expression:

[\P{M}\P{Lm}]      # Match a character that is NOT a character intended to be combined with another character or a special character that is used like a letter
(?:                # Start a group for the combining characters:
  (?:                # Start a group for tied characters:
    [\u035C\u0361]      # Match an under- or over- tie bar...
    \P{M}\p{M}*         # ...followed by another grapheme (in the simplified sense)
  )                  # (End the tied characters group)
  |\p{M}             # OR a character intended to be combined with another character
  |\p{Lm}            # OR a special character that is used like a letter
)*                 # Match the combining characters group zero or more times.

We could probably also create our own IEnumerator<string> using CharUnicodeInfo.GetUnicodeCategory to regain our performace, but that seems like too much work to me and extra code to maintain. (Anyone else want to have a go?) Regexes are made for this.

Answer 2

I'm not sure that's exactly what you're looking for, but isn't your question related to Unicode normalization ?

When a string is normalized to Unicode Form C (which is the default form), diacritics and the characters they modify are combined, so if you enumerate the characters you will get the base and modifier characters together.

When it is normalized to Form D, base and modifier characters are separated, and returned separately in an enumeration.

See the String.Normalize method for details