Just getting started with Lucene.Net. I indexed 100,000 rows using standard analyzer, ran some test queries, and noticed plural queries don't return results if the original term was singular. I understand snowball analyzer adds stemming support, which sounds nice. However, I'm wondering if there are any drawbacks to gong with snowball over standard? Am I losing anything by going with it? Are there any other analyzers out there to consider?
+2
A:
Yes, by using a stemmer such as Snowball, you are losing information about the original form of your text. Sometimes this will be useful, sometimes not.
For example, Snowball will stem "organization" into "organ", so a search for "organization" will return results with "organ", without any scoring penalty.
Whether or not this is appropriate to you depends on your content, and on the type of queries you are supporting (for example, are the searches very basic, or are users very sophisticated and using your search to accurately filter down the results). You may also want to look into less aggressive stemmers, such as KStem.
Avi
2010-10-06 17:52:04
I just figured out you can also do a fuzzy search like this "kangaroos~" that will return singular versions of the word as well, although it seems to take a bit longer to process the query.
alchemical
2010-10-06 19:02:39
@alchemical: I would really recommend against doing that. ~ is a very slow operator, and if your user does stuff like search for a phrase you're kinda screwed. Why is it so bad if you "kangaroos" is stored as "kangaroo"?
Xodarap
2010-10-06 21:49:00
OK, that's good to know -- to use KStem do you need Solr? Do you need to work with Lucene source code to integrate it in?
alchemical
2010-10-06 21:59:42
+1
A:
I just finished an analyzer that performs lemmatization. That's similar to stemming, except that it uses context to determine a word's type (noun, verb, etc.) and uses that information to derive the stem. It also keeps the original form of the word in the index. Maybe my library can be of use to you. It requires Lucene Java, though, and I'm not aware of any C#/.NET lemmatizers.
larsmans
2010-10-07 10:25:19
A:
The snowball analyzer will increase your recall, because it is much more aggressive than standart analyzer. So you need to evaluate your search results to see if for your data you need to increase recall or precision.
Skarab
2010-10-10 11:08:16