Is there a fast, accurate Highlighter for Lucene?

Question

I've been using the (Java) Highlighter for Lucene (in the Sandbox package) for some time. However, this isn't really very accurate when it comes to matching the correct terms in search results - it works well for simple queries, for example searching for two separate words will highlight both code fragments in the results.

However, it doesn't act well with more complicated queries. In the simplest case, phrase queries such as "Stack Overflow" will match all occurrences of Stack or Overflow in the highlighting, which gives the impression to the user that it isn't working very well.

I tried applying the fix here but that came with a lot of performance caveats, and at the end of the day was just plain unusable. The performance is especially an issue on wildcard queries. This is due to the way that the highlighting works; instead of just working on the querystring and the text it parses it as Lucene would and then looks for all the matches that Lucene has made; unfortunately this means that for certain wildcard queries it can be looking for matches to 2000+ clauses on large documents, and it's simply not fast enough.

Is there any faster implementation of an accurate highlighter?

Answer 1

You could look into using Solr. http://lucene.apache.org/solr

Solr is a sort of generic search application that uses Lucene and supports highlighting. It's possible that the highlighting in Solr is usable as an API outside of Solr. You could also look at how Solr does it for inspiration.

Answer 2

I've been reading on the subject and came across spanQuery which would return to you the span of the matched term or terms in the field that matched.

Answer 3

There is a new faster highlighter (needs to be patched in but will be part of release 2.9)

https://issues.apache.org/jira/browse/LUCENE-1522

and a back-reference to this question