I have a dataset consisting of roughly 10000 5 field records. I need to be able to retrieve a record based on the value in one field, which is a unique string.
Each record will likely only be accessed once or twice.
The application accessing the data uses Castle with ActiveRecord, backed onto SQL Server 2005. I recognize that at some point, indexing and retrieval will become faster and more efficent from the DB. But, ActiveRecord places an overhead on the existing database retrieval overhead.
At what point is it more efficent to load the data into the database (once off) and access from ActiveRecord, than to load it from CSV into memory when the web application first boots, or read it from the file as needed?