Running Long Process: Indexing 5GB docs with Lucene

Question

Situation:I have an ASP .NET application that will search through docs using Lucene. I want to run the initial indexing (the index will be incremental after the initial run so there wont be need to index the whole directory again in future). Currently, I have about 5GB of docs (45000files).

Problem: My application times out before completing the process. I have altered the TimeOut like this:

HttpContext.Current.Server.ScriptTimeout = 200000;

but it still does not complete the process.

How can I run the index?

Answer 1

You shouldn't run these from ASP.NET. Create a service that does the work for you, and kick it off via MSMQ, a "work_queue" table in the database, or whatever makes sense in your scenario.

Web Requests are intended to perform work quickly and immediately return, rather than do batch processing.