Faster way to get distinct values from Lucene Query

Question

Currently I do like this:

IndexSearcher searcher = new IndexSearcher(lucenePath);
Hits hits = searcher.Search(query);
Document doc;
List<string> companyNames = new List<string>();

for (int i = 0; i < hits.Length(); i++)
{
    doc = hits.Doc(i);
    companyNames.Add(doc.Get("companyName"));
}
searcher.Close();

companyNames = companyNames.Distinct<string>().Skip(offSet ?? 0).ToList();
return companyNames.Take(count??companyNames.Count()).ToList();

As you can see, I first collect ALL the fields (several thousands) and then distinct them, possibly skip some and take some out.

I feel like there should be a better way to do this.

Answer 1

I'm not sure there is, honestly, as Lucene doesn't provide 'distinct' functionality. I believe with SOLR you can use a facet search to achieve this, but if you want this in Lucene, you'd have to write some sort of facet functionality yourself. So as long as you don't run into any performance issues, you should be ok this way.

Answer 2

Tying this question to an earlier question of yours (re: "Too many clauses"), I think you should definitely be looking at term enumeration from the index reader. Cache the results (I used a sorted dictionary keyed on the field name, with a list of terms as the data, to a max of 100 terms per field) until the index reader becomes invalid and away you go.

Or perhaps I should say, that when faced with a similar problem to yours, that's what I did.

Hope this helps,