Short answer
You application pool or website is being shutdown too soon. Extend the idle timeout on the site, extend the application pool lifetime for the pool running the site. Raise the memory allocation and request limits.
Full answer
If you want to know when and why something is being removed from the cache, you need to log the item removal using the CacheItemRemovedCallback option on the insertion... Then you can log the reason using the CacheItemRemovedReason argument. You can thus log the reason as one of the four listed reasons:
- Removed The item is removed from the cache by a Remove method call or by an Insert method call that specified the same key.
- Expired The item is removed from the cache because it expired.
- Underused The item is removed from the cache because the system removed it to free memory.
- DependencyChanged The item is removed from the cache because the cache dependency associated with it changed.
Typically, you will find Expired and Underused being the reasons for things that don't have explict Remove calls made against the cache and don't have dependencies.
You will likely find out, while tracing through this fun stuff, that your items are not being expired or underused. Rather, I suspect you'll find that the AppDomain is getting unloaded.
One way this can happen due to the web.config (or bin directory, or .aspx, etc.) files getting changed. For more information as to when this occurs see the Application Restarts section of this page. When that happens, the currently pending requests are drained, the cache emptied, and the AppDomain unloaded. You can detect this situation by checking the AppDomain.IsFinalizingForUnload and logging that during the callback.
Another reason for the AppDomain to recycle is when IIS decides to recycle the AppPool for any of the reasons it has been configured with. Examples of that are xxx memory has been allocated over the lifetime, yyy seconds of runtime for the AppPool, ttt scheduled recycle time, or iiii idle time (no requests incoming). For further details check this article for IIS6 or this article for IIS7