How to index a web site.

Question

I'm asking on behalf of somebody, so I don't have too many details.

What options are available for indexing site content in an ASP.NET web site? I suspect SQL Server's Full Text index may be used if the page content is stored in the database. How would I index dynamic and static content if that content isn't stored in the DB, but in html and aspx pages themselves?

Answer 1

You can use a web crawler to crawl that site and add the content to a database which then is full text indexed. There are a number of web crawlers out there.

Answer 2

http://www.sphinxsearch.com/

Answer 3

Lucene is a well known open source tool that would help you here. The main branch is Java based but there is a .Net port too.

Main site: http://lucene.apache.org/ .Net port: http://lucene.apache.org/lucene.net/