How to merge matches from two distinct (not sharded) Lucene Indexes.

Question

I have two separate indexes holding different fields that together contain all the searchable fields for an index. For example the first index holds the indexed text for all documents, and the second holds tags for each and every document.

Note the example below is a bit wonky as I've changed the names of the entities. Index1: text document-id

Index2: tag-name: "very important" user: "Fred's id"

I would like to keep the indexes separate as it seems wasteful to continually update a single index whenever a user adds/removes a tag.

So far I think I might need to process the two search results and merge them manually (in code).Any other suggestions ?

I do not want to merge separate/sharded indexes.

Answer 1

Seems like you need to merge the indexes in code. If I understand correctly, when searching for a term, there can be either matches to document text or to tags, and each tag is indexed with its relevant document ids. You will then have two hit lists to merge. As tags and full text are very different entities, you will need some weighting (maybe as field boosts during retrieval) to reach a good ranking. Thus, you can merge the tag hit and full text hit for document k using a formula like:

score(k) = a*tagscore(k)+b*fulltextscore(k)

Where a and b will be empirically determined coefficients.

For a much more detailed discussion, see Grant Ingersoll's findability and debugging relevance issues in search papers.

Answer 2

The main problem with this approach has to do with ranking of documents because the default algorithm (and probably most custom algorithms, with a few exceptions) are based on term frequency and inverse document frequency.

In other words, the Scorer needs to know how many times a term appears within a document, as well as how many other documents contain the term. This information is stored for each term in the index, but not the aggregate across multiple indexes.

The common solution to this problem is a two-phased approach. First, the query is run against each index to determine how many documents contain each term. Next, the results are aggregated and the query is run again, but this time the inverse document frequency is sent along with it.

As you can imagine, this won't perform as well as running a query against a single index, but since nothing is free, I suppose that is the trade-off to storing documents across multiple indexes.