Caching Large Datasets

Question

I'm working on an ASP.NET application that has the following requirements:

1. Once every 15 minutes perform a fairly expensive query of around 20,000 items (not from a database). Has around 10 columns, all short strings, except for one which stores up to 3000 char (usually a lot less).
2. Process the resulting DataTable with various sorting and filtering, then store the top 100 items in additional DataTables.
3. Display this information as a form of aggregation to potentially 10,000's of people.

It seems to me that this is an excellent candidate for caching (System.Web.Caching) especially given we may wish to support additional "on the fly" filtering. For example: Filtering the table down to rows only relevant to a specific user.

However, before getting started I would like to understand:

1. If there are any best practices around storing such a large DataTable in Cache.
2. If anyone has any experience they are able to share? Large tables you have stored?
3. Any traps to be careful of along the way?

Thanks in advance.

Answer 1

Have you looked at the Microsoft Enterprise Library Caching Application Block

It's build around best practises and very easy to use

http://msdn.microsoft.com/en-us/library/cc309103.aspx

Answer 2

It is pretty straight forward, but you may want to keep an eye on what is actually cached. Rick Strahl has an interesting post on how the cache was actually empty due to memory pressures.

Answer 3

Steve Smith won a competition at a Microsoft event with one of his caching solutions. Here is a post from his blog. Here is the DNR TV episode where he reviews his techniques.

Answer 4

Presumably the some rows in the data "table" are updated between query runs. So you would need to refresh the cache before every run whichs will give you zero improvement over just reading the file.

The file system operates a pretty efficient cache anyway so why re-invent the wheel.

A better approach may be to re-write your "query" so you get everything you need in a single pass of the file.