What is the real difference between INDEX.TOKENIZER vs INDEX.ANALYZER in Lucene 2.9?

Question

With lucene 2.9.1, INDEX.TOKENIZED is deprecated. The documentation says it is just renamed to ANALYZER, but I don't think the meaning has stayed the same. I have an existing app based on 2.3 and I'm upgrading to 2.9, but the expected behavior seems to have changed.

Anyone know any more details about INDEX.TOKENIZER vs INDEX.ANALYZER?

Answer 1

I assume you refer to the Field.Index fields ANALYZED and TOKENIZED?

It is true that the TOKENIZED Field has been deprecated. This was the case already with the 2.4.

The Field.Index.ANALYZED is equal to the old Field.Index.TOKENIZED. Could you show how your results deviate from the behaviour you expect?