What are my options for a search engine database on windows

Question

I have a project to create a high traffic search engine similar to altavista.com. The windows .NET C# will be used for the project. I am looking for a good search engine database that can handle a very high load. I have taken a look at lucene and sql server 2008. I have read that lucene tends to get corrupt when the load is very high. So, I am considering sql server 2008 but I am uncertain sql server can handle a very high load over gigabytes of data.

The database will be given a chunk of data and needs to fetch similar data elsewhere in the table. it will act just the similar questions area on stackoverflow. It will need to search over millions of rows. Is sql server 2008 and lucene my only options

Answer 1

I have used Lucene sucessfully with high volume datas and i dint face data corruption problems.

But if you are worried about the standalone lucene, you can try apache solr instead, its a open-source search server based on the Lucene

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites.

There is a .net client for solr in google code solr.net.. You can try this

Cheers