php, search in chinese and russian

Question

Eventually, I'm creating a website on three languages: english, russian and chinese. I hope that if I use UTF-8 in application and database, there won't be any problems with input-output (will there?)

But the most frightening part of it is a search. It should be cool enough. It should be full-text, it should index, etc. I hope it will understand morfology, use stemming, etc.

First, I've looked at Zend_Search_Lucene, but as I realised from http://framework.zend.com/issues/browse/ZF/component/10021 it has problems with Chinese. :(

Now I'm thinking about Sphinx. It supports both English and Russian stemming. I'm not sure how good is it with Chinese and I have no idea how hard will it be for me to add support for it. http://www.sphinxsearch.com/forum/view.html?id=1554 is a silver lining but, as not experienced Sphinx user, I don't think I understand what is said there.

---

So,

does anyone have any experience in such 'language-agnostic' search and can share it with me, please?

and can you give me something to test the search. As a native Russian speaker with some basic knowledge of English I can test both Russian and English searches by myself, but I don't even know about which parts of this Chinese pics are words. Please, give me some Chinese strings to put them into index and some queries with expected results!

Answer 1

From Xapian docs :

Xapian uses the Snowball Stemming Algorithms. At present, these support Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, and Turkish. There are also implementations of Lovins' English stemmer, Porter's original English stemmer, the Kraaij-Pohlmann Dutch stemmer, and a variation of the German stemmer which normalises umlauts.

or some of the world's languages, Chinese for example, the concept of stemming is not applicable, but it is certainly meaningful for the many languages of the Indo-European group.

http://xapian.org/docs/stemming.html

Answer 2

Isn't Google User Search enough for your needs? What exactly don't you like in it?

Answer 3

Hi!

Ideographic characters in languages such as Chinese or Japanese require two terminal character positions, so you will have problems with UTF8 and you should use UTF16 instead.

Apart from that, any search engine supporting UTF16 and your requirements (e.g. stemming) should work fine - that is, if you like Sphinx, go for it!