Caching strategies for Windows end-user applications?

Question

I'm working on what is essentially the runtime for a large administrative application. The actual logic that is being executed, as well as the screens being shown and the data operated upon is stored in a central database. In order to improve performance, the runtime keeps data queried from the database in various caches.

However, it is not always clear how these caches should be managed. Currently, some caches are flushed whenever the runtime goes idle, whereas other caches are never flushed, or only flushed if some configurable but arbitrary limit is reached. We'd obviously want to keep as much data as possible in memory, yet I'm unsure how to do this in a way that plays nicely with Citrix, something that's very important to our customers.

I've been looking into using a resource notification (CreateMemoryResourceNotification()) and flushing caches if it signals that memory is running low, but I'm afraid that using just that would make things behave very badly when running 20+ instances under Citrix, with one instance gobbling up all memory and the rest constantly flushing their caches.

I could set hard limits on cache size with CreateJobObject() and friends, but that could cause the runtime to fail with out-of-memory errors should an instance have a legitimate need for a lot of memory.

I could prevent such problems by using a separate heap for cached data, but there's not a clear separation between cached and non-cached data, so that seems awfully fragile.

TL;DR: anyone got any good ideas for managing in-memory caches under Windows?

Answer 1

I've been looking into using a resource notification (CreateMemoryResourceNotification()) and flushing caches if it signals that memory is running low, but I'm afraid that using just that would make things behave very badly when running 20+ instances under Citrix, with one instance gobbling up all memory and the rest constantly flushing their caches. I could set hard limits on cache size with CreateJobObject() and friends, but that could cause the runtime to fail with out-of-memory errors should an instance have a legitimate need for a lot of memory.

Can't you make a hybrid solution of some kind, so that the runtime tries to keep its cache limited to a fixed size, but with the possibility to grow bigger if there is a legitimate need to do so and then try to shrink the cache to a reasonable size if the occasion is there?

Preventing one instance from gobbling up all memory while the others are repeatedly flushing their caches can maybe avoided by distributing the memory resource notification to all instances when it arrives. This way they all take a good look at their caches when one instance gets the notification.

And last, of course sometimes a trade-off between performance and memory usage has to be made. Here again, if the instances can communicate in some way, they may be able to adjust their maximum cache size based on the number of instances and the amount of memory available on the machine they run on. This way, if more instances are started, they all give in a little bit to accommodate the newcomer, without the risk of overloading the memory of the server.

What strategy are you going to use to determine what needs to be cached? Are you going to keep a last-used timestamp and flushing old items when room needs to be made for new ones?