How to detect and debug stale cache entries?

Question

I am using memcached with PHP trying for heavy caching to avoid db-reads. I am invalidating the cache on an update (application invalidates). But, stale cache data is becoming a big issue. Mostly, this is due to bug in invalidation (invalidates wrong key OR forgetting to invalidate the cache entry on an UPDATE).

Is there any good method to detect/debug these type of bugs during development/production?

Answer 1

You can use optimistic concurrency control, which involves adding one more column to a table (either timestamp or equivalent, or just a plain int) and using it when doing updates. Here's how you might use it (in case of plain int being a "version tag"):

update BlogPost set PublishedOn = :publishedOn, VersionTag = VersionTag + 1
where ID = :id and VersionTag = :versionTag

And assuming timestamp is updated automatically by your DBMS, this is how it's done with timestamps:

update BlogPost set PublishedOn = :publishedOn
where ID = :id and Timestamp = :timestamp

Answer 2

The best thing to do is to put a wrapper function around mysql_query. Inside of that, just check if the query is an update or a delete to a cached table and parse out the keys that are being changed. This won't take long to write, is easy to test, and will prevent you from ever forgetting a cache invalidation again.

Answer 3

You could add debug code (which is disabled during production) in your application, which does two fetches - one from memcached and one from the DB, compares them and throws an exception / writes the error to the logfile if they are different. This way (with some loss of performance) you can quickly observe old data.

Of course, this can be further refined:

• Add a timestamp field and do not throw an error if the timestamps are closer than a defined value (maybe the process didn't have time to update the cache)
• Add a field in the DB which specifies the update source (script / line number). This could help in tracking down the part which misses the cache invalidation part