Finding the position of search hits from Lucene

Question

With Lucene, what would be the recommended approach for locating matches in search results?

More specifically, suppose index documents have a field "fullText" which stores the plain-text content of some document. Furthermore, assume that for one of these documents the content is "The quick brown fox jumps over the lazy dog". Next a search is performed for "fox dog". Obviously, the document would be a hit.

In this scenario, can Lucene be used to provide something like the matching regions for found document? So for this scenario I would like to produce something like:

[{match: "fox", startIndex: 10, length: 3},
 {match: "dog", startIndex: 34, length: 3}]

I suspect that it could be implemented by what's provided in the org.apache.lucene.search.highlight package. I'm not sure about the overall approach though...

Answer 1

Hit highlighting is a pretty common thing - check out this tutorial. http://www.cocooncenter.org/articles/lucene.html

Answer 2

TermFreqVector is what I used. Here is a working demo, that prints both the term positions, and the starting and ending term indexes:

    # public class Search {  
#       
#     public static void main(String[] args) throws IOException, ParseException{  
#     
#         Search s = new Search();  
#         s.doSearch(args[0], args[1]);  
#     }  
#       
#     Search(){}  
#       
#     public void doSearch(String db, String querystr) throws IOException, ParseException{  
#           
#         // 1. Specify the analyzer for tokenizing text.  
#         //    The same analyzer should be used as was used for indexing  
#         StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);  
#       
#         Directory index = FSDirectory.open(new File(db));  
#       
#         // 2. query  
#         Query q = new QueryParser(Version.LUCENE_CURRENT, "contents", analyzer).parse(querystr);  
#       
#         // 3. search  
#         int hitsPerPage = 10;  
#         IndexSearcher searcher = new IndexSearcher(index, true);  
#         IndexReader reader = IndexReader.open(index, true);  
#         searcher.setDefaultFieldSortScoring(true, false);  
#         TopScoreDocCollector collector = TopScoreDocCollector.create(hitsPerPage, true);  
#         searcher.search(q, collector);  
#         ScoreDoc[] hits = collector.topDocs().scoreDocs;  
#           
#         // 4. display term positions, and term indexes   
#         System.out.println("Found " + hits.length + " hits.");  
#         for(int i=0;i<hits.length;++i) {  
#               
#             int docId = hits[i].doc;  
#             TermFreqVector tfvector = reader.getTermFreqVector(docId, "contents");  
#             TermPositionVector tpvector = (TermPositionVector)tfvector;  
#             // this part works only if there is one term in the query string,  
#             // otherwise you will have to iterate this section over the query terms.  
#             int termidx = tfvector.indexOf(querystr);  
#             int[] termposx = tpvector.getTermPositions(termidx);  
#             TermVectorOffsetInfo[] tvoffsetinfo = tpvector.getOffsets(termidx);  
#               
#             for (int j=0;j<termposx.length;j++) {  
#                 System.out.println("termpos : "+termposx[j]);  
#             }  
#             for (int j=0;j<tvoffsetinfo.length;j++) {  
#                 int offsetStart = tvoffsetinfo[j].getStartOffset();  
#                 int offsetEnd = tvoffsetinfo[j].getEndOffset();  
#                 System.out.println("offsets : "+offsetStart+" "+offsetEnd);  
#             }  
#               
#             // print some info about where the hit was found...  
#             Document d = searcher.doc(docId);  
#             System.out.println((i + 1) + ". " + d.get("path"));  
#         }  
#       
#         // searcher can only be closed when there  
#         // is no need to access the documents any more.   
#         searcher.close();  
#     }  
#       
# }