ASP.NET search indexing building strategy

Question

This is what I'm planning to do and I'd appreciate anyone's input:

I've built a forum in Asp.net MVC and now want to add Lucene.Net for search. My plan is to run an index builder thread every 5-10 minutes to update the search index with the changes made to the each discussion.

The way it will work is I keep the date and time for the last run of the index builder thread in the search index. Then on every execution of the index builder, I read this date back from the search, then index any changes since that date and time. Once I'm done I then update the last run entry.

Is this way good? Can someone suggest a better way to incrementally index changes in a forum app?

Answer 1

You will need to maintain a timer... and if the indexing operation doesn't stop in 5 minutes another one will start indexing the same changes so you'll have to check for such condition as well.

A slightly better way is to simply use a dedicated indexing Thread that stays alive. This Thread will fetch changes from the last run and process them as you describe, but it will not wait. After the index operation finishes, it'll re-start itself right away continually indexing as items are in.

If there are no more items to index, the Thread will then sleeps for 5 minutes (and then re-check for changes again when it wakes up).

This way you can be sure that there will only be one client at a time modifying the indexes. It'll never take up a lot of CPU as might be the case if you mismanaged the timer somehow or you suddenly got a flood of posts, and will scale as your forum grows without needing to adjust the indexing interval every now and then.

You will need to monitor the Thread's health though.