what are the fastest/Popular search technologies

Question

What are the fastest search technologies apart from Relational DB Searches ?

I have a collection of Text Files from varied sources (Banks/Ledgers/Stock Markets). Each line in these Text Files is a Record. Each line can further be parsed into some DB Columns(Stock Name/Date of purchase/Owner/...). It is not necessary that each line has all the fields. If some text is not mathcing any of the fields, I store it as it is in a seperate column(columnA). If the parser fails to find any column at all I just store the Line in columnA.

I than do a DB search based on user Inputs. I find that the search becomes slow after a few million records. I want to move to an alternative storage like File System or some other Indexed storage.

What do you think would be the fastest storage technology for this kind of requirement ?

I am currently using SQLite DB. Since I am about to rewrite the entire code, I will be more than happy if any other storage technology can give me faster results.

Answer 1

Relational databases aren't great at text searching. Perhaps you could try something more suited to the task like solr. That said, it seems like you're putting a lot of data in SQLite (it might be a bit "lite" for your purpose). If you switched to something like MySQL or Postgres you might get the speed you need without much of a rewrite.

Answer 2

1. Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
2. Lucene.Net is a source code, class-per-class, API-per-API and algorithmatic port of the Java Lucene search engine to the C# and .NET platform utilizing Microsoft .NET Framework.