Does Sphinx handle content in asian languages well?

Question

I'm thinking about using Sphinx as a search engine for my site. But since I have a lot of Korean content, and other languages like Chinese and Thai may follow, I wonder how well Sphinx can handle this type of content.

Answer 1

Sphinx works well for UTF-8 characters (which includes Korean I believe), but you'll have to include a list of UTF-8 characters codes to index in your sphinx config file.

This is how my charset_table variable looks like in sphinx config, to add all kinds of characters from European languages:

charset_table       = 0..9, A..Z, U+00C0..U+00DE, U+0100, U+0102, U+0104, U+0106, U+0108, U+010A, U+010C, U+010E, U+0110, U+0112, U+0114, U+0116, U+0118, U+011A, U+011C, U+011E, U+0120, U+0122, U+0124, U+0126, U+0128, U+012A, U+012C, U+012E, U+0130, U+0132, U+0134, U+0136, U+0139, U+013B, U+013D, U+013F, U+0141, U+0143, U+0145, U+0147, U+014A, U+014C, U+014E, U+0150, U+0152, U+0154, U+0156, U+0158, U+015A, U+015C, U+015E, U+0160, U+0162, U+0164, U+0166, U+0168, U+016A, U+016C, U+016E, U+0170, U+0172, U+0174, U+0176, U+0178, U+0179, U+017B, U+017D, a..z, U+00DF..U+00F6, U+00F8..U+00FF, U+0101, U+0103, U+0105, U+0107, U+0109, U+010B, U+010D, U+010F, U+0111, U+0113, U+0115, U+0117, U+0119, U+011B, U+011D, U+011F, U+0121, U+0123, U+0125, U+0127, U+0129, U+012B, U+012D, U+012F, U+0131, U+0133, U+0135, U+0137, U+0138, U+013A, U+013C, U+013E, U+0140, U+0142, U+0144, U+0146, U+0148, U+0149, U+014B, U+014D, U+014F, U+0151, U+0153, U+0155, U+0157, U+0159, U+015B, U+015D, U+015F, U+0161, U+0163, U+0165, U+0167, U+0169, U+016B, U+016D, U+016F, U+0171, U+0173, U+0175, U+0177, U+017A, U+017C, U+017E, U+017F, U+0027

Answer 2

I am using Sphinx to search CJK charcters (Chinese, Japanese, and Korean), what you need to do is to add the following lines in your index block of your configuration file.

index test {
  ...
  charset_type = utf-8
  ngram_len = 1
  ngram_chars = U+3000..U+2FA1F
}