How to remove deleted entries from the main sphinx index?

Question

I have a site that utilizes the main/delta indexing scheme for sphinx. main gets rebuilt daily, delta is rebuilt every 5 minutes. This works well for indexing newly submitted items.

The problem is, items needed to be dropped from the index just as frequently as they are added, and typically the dropped items are older, so they already reside in the main index. So after item is deleted, they still appear in the search results for up to 24 hours (until 1am when main is rebuilt).

How can I solve this issue?

Answer 1

Sphinx recently introduced a realtime indexes feature, where you can add, update, and delete indexes on the fly. However, it's only available in 1.10, and still seems pretty raw.

http://www.sphinxsearch.com/docs/current.html#rt-indexes

Alternatively, you could full re-index more often. Every 24 hours seems kind of long, if you have a lot of deletions. As a last-ditch effort, you could always check back in your application that the IDs being returned do indeed still exist, and then filter those out.