views:

984

answers:

3

The title asks it all. The content on the site I'm building wont change very quickly at all and so Memcache could potentially store data for months except for when I put up an update. Is there a way to make it clear the cache every time I deploy the site? I'm using the Python runtime.

Update 1

Using jldupont's answer I put the following code in my main request handling script...

Update 2

I've switched to the method mentioned by Koen Bok in the selected answer's comments and prefixed all my memcache keys with os.environ['CURRENT_VERSION_ID']/ with the helpful code in the answer's 2nd update. This solution seems to be much more elegant than the function I posted before.

+10  A: 

Have you tried flush_all() function? Docs here. You'll need a bit of logic & state to detect a new deployment or have a special script to perform the flushing.

Updated: look at the absolute path of one of your script: this changes on every deployment. You can use http://shell.appspot.com/ to experiment:

  import sys
  sys.path

['/base/python_dist/lib/python25.zip', '/base/python_lib/versions/third_party/django-0.96', '/base/python_dist/lib/python2.5/', '/base/python_dist/lib/python2.5/plat-linux2', '/base/python_dist/lib/python2.5/lib-tk', '/base/python_dist/lib/python2.5/lib-dynload', '/base/python_lib/versions/1', '/base/data/home/apps/shell/1.335852500710379686/']

Look at the line with /shell/1.335852500710379686/.

So, just keep a snapshot (in memcache ;-) of this deployment state variable and compare in order to effect a flushing action.

Updated 2: as suggested by @Koen Bok, the environment variable *CURRENT_VERSION_ID* can be used also (part of the absolute path to script files also).

 import os
 os.environ["CURRENT_VERSION_ID"]
jldupont
Yeah, I'm aware of that function. How would I detect a new deployment, that's my main problem?
donut
you can look at the absolute path of one of your script: this will change on every deployment.
jldupont
You could use remote_api_shell.py to run flush_all() after you deploy. It's manual, but easy.
Adam Crossland
This is frickin' clever. Moar upvotes!
Jonathan Feinberg
Thanks a lot for elaborating. I posted the resulting code in my original question. Works better than a charm!
donut
My pleasure! Cheers.
jldupont
Why not just take CURRENT_VERSION_ID from the environment variables? And prefixing the cache keys is faster then adding one extra memcache lookup for each request with the above method.
Koen Bok
A: 

You could just create an admin-only path that would flush the cache when it's accessed.

donut
Well, now that's a good idea! But it would best if this could be automated somehow like jdupont hints at.
donut
It could be automated by hacking appcfg.py to include a call to the cache flushing URL.
Adam Crossland
Eek, forking the SDK! I'd rather call appcfg.py from a script to upload, and have that script do whatever else, than modify it. Especially since I already only upload from a makefile anyway...
Steve Jessop
That would work, too, Steve.
Adam Crossland
+1  A: 

When creating keys for your cached values, include the version of the file that is doing the cache gets/sets in the key. That way, when a new version of the file exists, it will no longer reference the old versions in the cache - they will be left to expire out on their own.

We use CVS and java, so we declare this variable in each file that will do caching:

private static final String CVS_REVISION = "$Revision $";

When you check that file out, you'll get something like this:

private static final String CVS_REVISION = "$Revision: 1.15 $";

You can adapt for your language and version control system if not CVS. Remember to encode special characters out of your keys. We've found that URL Encoding key values works well for memcached.

Matt
I use git and am not familiar enough with it to know if this is possible. It does seem like it may be a more wholistic solution. But jdupont's solution will work without help from a version control system and more accurately answers the question.
donut
I do exactly this, but I take the version from the appengine environment variables. Works great.
Koen Bok
Glad you found a good solution. This idea works well for us, but the other solution looks better for you.
Matt