template fragment caching doesn't seem to work for some custom template tags

Question

I've been implementing caching in my django application, and used per view caching via the cache API and template fragment caching. On some of my pages I use a custom django template tag, this tag is provided via a third party developer, it takes some arguments in its template tags, and then make a request to a remote server, gets the response back over XML, and then renders the result in my page. Great - I thought I could easily cache this using fragment caching, so I :

{% load cache %}
{% cache 500 request.user.username %}
{% load third party custom tags %}
{% expensive custom tag set that gets stuff from a third party server via xml %}
{{ some.stuff}}
{% endcache %}

Trouble is no matter what I do, the requests still get fired off to that remote server, it seems Django doesn't like to cache these custom template tags. I know memcached is working great, for other views and templates it all works just fine. Am I doing something that is incompatible with the fragment caching? Is there a way round it?

Answer 1

Have you tried to use a different name for the cache fragment? There could be a problem with using request.user.username for a couple of reasons:

• If a user is not signed in, request.user.username could be empty, resulting in a non-named cache fragment

• If a user is signed in, this will call the 3rd party template tag at least once for each user every 3 mintues

Maybe it's worth trying to rename the cache fragment name to test:

{% cache 500 customxml %}

I'd also try loading of the 3rd party template tag outside the cache tag like so:

{% load cache third_party_custom_tags %}
{% cache 500 request.user.username %}
{% expensive custom tag set that gets stuff from a third party server via xml %}
{{ some.stuff}}
{% endcache %}

What I'm not sure of is if the cache framework caches the results of a template tag. If that doesn't work, I'd take a look at what the template tag is doing under the hood and re-implement the template tag using Django's low-level cache.

Answer 2

I think the problem is the custom tag, as you suggested.

I disagree that the request.user.username is a problem, as the documentation for the subject actually gives that as an example, and I have used it with internal caching (number of posts, for instance), in testing, and it worked fine.

The low level cache is potentially useful, but I would take a look at your custom tag to see what wouldn't be caching. Without the code it's hard to guess, but my guess would be something like the time, or some other variable, is being returned whic his causing it to force an update (if the XML pulls any data that changes Django may force an update, depending on other settings). I've had mixed results with Django's caching, so I would have a look at your XML feed(s) to see if it's causing anything to stop caching.

Answer 3

I don't think this has anything to do with the custom tag.

We endded up rewriting the Django caching tag because we needed more control than was possible with the one that was supplied. You might make a copy of it yourself and stick some debugging print statements into it. In particular, check the filename (assuming you are caching to files) and see what is being generated. It could be that it is changing when it shouldn't (for some unknown reason) and that would mean that it is always needing to re-render then enclosed block.

Look in django/templatetags/cache.py. It's only 63 lines of code.

Answer 4

If the template fragment you're trying to cache can't be pickled, memcached won't be able to store it and will raise an exception. From what I can gather, exceptions generated when rendering Django templates are suppressed. Since your custom tag is doing HTTP requests, maybe socket objects (which can't be pickled) are getting stored to the template fragment somehow.

If this is the case, the only way around it I can think of would be to modify the custom tag to get rid of any leftover socket objects.