Enabling soundex/metaphone for non-English characters

Question

Hi all,

I've been studying soundex, metaphone and other string search techniques the past few days, and in my understanding both algorithms work well in handling non-English words transliterated to English.

However the requirement that I have would be for such search to work in the original, untransliterated languages, accomodating alphabets such as German, Norwegian, and even Cyrilic alphabets.

Are there any search algorithms capable of handling these alphabets completely? Or am I better off using third party full-text-search libraries such as Lucene? Consequently, the question then becomes 'does Lucene handle non-English alphabets?'

Answer 1

There are some good references on Wikipedia, starting from the Soundex article. I don't know whether there are existing libraries designed to handle such a wide variety of languages.

Answer 2

I'm not an expert in this area, but your requirements seem quite difficult to me. Soundex was specifically designed for English sounds as well as characters. I don't think it will perform well for non-English languages. See for example the responses to this related question.

Double-Metaphone is an attempt to deal with much more complex variations than Soundex or Metaphone, and was designed to handle irregularities in a range of languages. It might be sufficient for your needs. There is a list of library implementations on the linked page.

Support for other languages in Lucene is based on the concept of Analyzers. Lucene comes with a set of analyzers for different languages (although I couldn't find the default list), but the quality may be quite variable.