Lucene HTMLFormatter skipping last character

Question

I have this simple Lucene search code (Modified from http://www.lucenetutorial.com/lucene-in-5-minutes.html)

 class Program
    {
        static void Main(string[] args)
        {

            StandardAnalyzer analyzer = new StandardAnalyzer();
            Directory index = new RAMDirectory();
            IndexWriter w = new IndexWriter(index, analyzer, true,
                    IndexWriter.MaxFieldLength.UNLIMITED);
            addDoc(w, "Table 1 <table> content </table>");
            addDoc(w, "Table 2");
            addDoc(w, "<table> content </table>");
            addDoc(w, "The Art of Computer Science");
            w.Close();


            String querystr = "table";


            Query q = new QueryParser("title", analyzer).Parse(querystr);
            Lucene.Net.Search.IndexSearcher searcher = new
            Lucene.Net.Search.IndexSearcher(index);
            Hits hitsFound = searcher.Search(q);

            SimpleHTMLFormatter formatter = new SimpleHTMLFormatter("*", "*");

            Highlighter highlighter = null;
            highlighter = new Highlighter(formatter, new QueryScorer(searcher.Rewrite(q)));

            for (int i = 0; i < hitsFound.Length(); i++)
            {
                Console.WriteLine(highlighter.GetBestFragment(analyzer, "title", hitsFound.Doc(i).Get("title")));
             //   Console.WriteLine(hitsFound.Doc(i).Get("title"));
            }
            Console.ReadKey();



        }
        private static void addDoc(IndexWriter w, String value)
        {
            Document doc = new Document();
            doc.Add(new Field("title", value, Field.Store.YES, Field.Index.ANALYZED));
            w.AddDocument(doc);
        }
    }

The highlighted results always seem to skip the closing '>' of my last table tag. Any suggestions?

Answer 1

Lucene's highlighter, out of the box, is geared to handle plain text. It will work incorrectly if you try to highlight HTML or any mark-up text.

I recently ran into the same problem and found a solution in Solr's HTMLStripReader which skips the content in tags. The solution is outlined on my blog at following URL.

http://sigabrt.blogspot.com/2010/04/highlighting-query-in-entire-html.html

I could have posted the code here, but my solution is applicable for Lucene Java. For .Net, you have to find out equivalent of HTMLStripReader.

Answer 2

Solved. Apparently my Highlighter.Net version was archaic. Upgrading to 2.3.2.1 Solved the problem