tags:

views:

107

answers:

3

In django, I wrote a view that simply returns a file, and now I am having problems because memcache is trying to cache that view, and in it's words, "TypeError: can't pickle file objects".

Since I actually do need to return files with this view (I've essentially made a file-based cache for this view), what I need to do is somehow make it so memcache can't or won't try to cache the view.

I figure this can be done in two ways. First, block the view from being cached (a decorator would make sense here), and second, block the URL from being cached.

Neither seems to be possible, and nobody else seems to have run into this problem, at least not on the public interwebs. Help?

Update: I've tried the @never_cache decorator, and even thought it was working, but while that sets the headers so other people won't cache things, my local machine still does.

+4  A: 

Returning a real, actual file object from a view sounds like something is wrong. I can see returning the contents of a file, feeding those contents into an HttpResponse object. If I understand you correctly, you're caching the results of this view into a file. Something like this:

def myview(request):
    file = open('somefile.txt','r')
    return file    # This isn't gonna work. You need to return an HttpRequest object.

I'm guessing that if you turned caching off entirely in settings.py, your "can't pickle a file object" would turn into a "view must return an http response object."

If I'm on the right track with what's going on, then here are a couple of ideas.

You mentioned you're making a file-based cache for this one view. You sure you want to do that instead of just using memcached?

If you really do want a file, then do something like:

def myview(request):
    file = open('somefile.txt','r')
    contents = file.read()
    resp = HttpRespnse()
    resp.write(contents)
    file.close()
    return resp

That will solve your "cannot pickle a file" problem.

Unoti
Thanks for the thoughtful comments. Yeah, I'm essentially doing what you're suggesting. And the reason I'm not using memcached for this is because it's for the sitemaps on the site, which take 20 seconds to generate (or so), and which would very quickly fill up memcached (I have about 600 sitemaps). By using a home-grown file-based cache, they get generated once, and then served consistently. If they ever change (they don't, usually), I just delete the files from disk manually, and all is well!
mlissner
Amazingly, this was ultimately the answer. I thought I had my cache working properly, but I missed one area where it was trying to simply return a file. Ugh! Kudos on the bounty.
mlissner
+2  A: 
from django.views.decorators.cache import never_cache

@never_cache
def myview(request):
    # ...

Documentation is here...

FallenAngel
That keeps *other* computers from caching things, but my local install of memcached still caches stuff...frustratingly.
mlissner
If you're asking about the standard Django cache middleware, this answer is correct. The Django cache framework is controlled by standard HTTP `Cache-Control` headers like the ones set by the `@never_cache` decorator.
jpwatts
That's great to know. I'm really having a tough time knowing if something has been cached, apparently.
mlissner
A: 

You probably did a per site cache, but what you want to do now is a per view cache. The first one is easier to implement, but is only meant for the case of 'just caching everything'. Because you want to choose for every view now, just switch to the fine grained approach. It is also very easy to use, but remember that sometimes you need to create a second view with the same contents, if you want to have the result sometimes cached and sometimes not, depending on the url.

So far to the answer to your question. But is that an answer to your problem? Why do you return files in a view? Normally static files like videos, pictures, css, flash games or whatever should be handled by the server itself (or even by a different server). And I guess, that is what you want to do in that view. Is that correct? The reason for not letting django do this is, because starting django and letting django do its thing also eats a lot of resoruces and time. You don't feel that, when you are the only user in your test environment. But when you want to scale to some thousand users or more, then this kind of stuff becomes very nasty. Also from a logical point of view it does not seem smart, to let a program handle files without changing them, when the normal job of the program is to generate or change HTML according to a state of your data and a user-request. It's like letting your accountant do the programming work. While he might be able to do it, you probably want somebody else do it and let the accountant take care of your books.

erikb
Very complete answer, but the reason I'm doing this is to implement a file-based cache on the sitemaps for the site, and a memcached cache everywhere else. Annoyingly, doing it myself was the only way I could keep my 600+ sitemaps from filling the cache all the time while having good response rates!
mlissner