Troubleshoot Java Lucene ignoring Field

Question

We're currently using Lucene 2.1.0 for our site search and we've hit a difficult problem: one of our index fields is being ignored during a targeted search. Here is the code for adding the field to a document in our index:

// Add market_local to index
contactDocument.add(
    new Field(
        "market_local"
        , StringUtils.objectToString(
            currClip.get(
                "market_local"
            )
        )
        , Field.Store.YES
        , Field.Index.UN_TOKENIZED 
    )
);

Running a query ( * ) against the index will return the following results:

Result 1:
    title: Foo Bar
    market_local: Local

Result 2:
    title: Bar Foo
    market_local: National

Running a targeted query:

+( market_local:Local )

won't find any results.

I realize this is a highly specific question, I'm just trying to get information on where to start debugging this issue, as I'm a Lucene newbie.

---

UPDATE

Installed Luke, checking out latest index... the Field *market_local* is available in searches, so if I execute something like:

market_local:Local

The search works correctly (in Luke). I'm going over our Analyzer code now, is there any way I could chalk this issue up to the fact that our search application is using Lucene 2.1.0 and the latest version of Luke is using 2.3.0?

Answer 1

For debugging Lucene, the best tool to use is Luke, which lets you poke around in the index itself to see what got indexed, carry out searches, etc. I recommend downloading it, pointing it at your index, and seeing what's in there.

Answer 2

The section on "Why am I getting no hits?" in the Lucene FAQ has some suggestions you might find useful. You're using Field.Index.UN_TOKENIZED, so no Analyzer will be used for indexing (I think). If you're using an Analyzer when you're searching then that might be the root of your problem - the indexing and searching Analyzers should be the same to make sure you get the right hits.

Answer 3

Another simple thing to do would be to use a debugger or logging statement to check the value of

StringUtils.objectToString(currClip.get("market_local"))

to make sure it is what you think it is.

Answer 4

Luke is bundled with Lucene, but you can tell Luke to use another version of Lucene. Say "lucene-core-2.1.0.jar" contains Lucene 2.1.0 that you want to use and "luke.jar" contains Luke with Lucene 2.3.0. Then you can start Luke with the following command.

java -classpath lucene-core-2.1.0.jar;luke.jar org.getopt.luke.Luke

(The trick is to put your version of Lucene before Luke on the classpath. Also, This is on Windows. On Unix, replace ";" with ":".)

As you can check in Luke,

+( market_local:Local )

gets rewritten to

market_local:Local

if the rewrite(IndexReader) method of the Query object is called. The two queries should be equivalent so there might be a bug in 2.1. If you have to use 2.1, you can try to manually call that method before passing the Query object to the IndexSearcher.