Sphinx Building Index Improvement

Question

Hey guys, I have a question about Sphinx. I use Sphinx to index the full-text searches for my sites, and it works like a dream. At this point in time it takes about 30 minutes to create the indexes for all my databases. This is fine as I only run the indexing script once every hour.

But the databases are getting bigger quickly, and soon I'm afraid the databases will be so big, it wont be able to do the process in 1 hour. Of course I can run it only once every 2 hours, but this is not ideal.

Now my question: Is sphinx rebuilding the entire indexes every time the script runs, or does it only add the newest items that were recently added to the database (since the last index run)?

My feeling says, that it totally rebuilds it.

In case that is true, is it also possible to ONLY the items that were not in it already? Could this make the indexing time a lot shorter?

Answer 1

See Delta index updates: http://sphinxsearch.com/docs/current.html#delta-updates

The idea is to maintain two indexes: main and delta. You only need to build the index for newly added/update contents into the delta index, and merge back to the main index periodically.