Can't get Beaker cache working

Question

Hi everyone,

I'm trying to use Beaker's caching library but I can't get it working.

Here's my test code.

class IndexHandler():
    @cache.cache('search_func', expire=300)
    def get_results(self, query):
        results = get_results(query)
        return results

    def get(self, query):
        results = self.get_results(query)
        return render_index(results=results)

I've tried the examples in Beaker's documentation but all I see is

<type 'exceptions.TypeError'> at /
can't pickle generator objects

Clearly I'm missing something but I couldn't find a solution.

By the way this problem occurs if the cache type is set to "file".

Answer 1

I see that the beaker docs do not mention this explicitly, but, clearly, the decorate function must pickle the arguments it's called with (to use as part of the key into the cache, to check if the entry is present and to add it later otherwise) -- and, generator objects are not pickleable, as the error message is telling you. This implies that query is a generator object, of course.

What you should be doing in order to use beaker or any other kind of cache is to pass around, instead of a query generator object, the (pickleable) parameters from which that query can be built -- strings, numbers, dicts, lists, tuples, etc, etc, composed in any way that is easy to for you to arrange and easy to build the query from "just in time" only within the function body of get_results. This way, the arguments will be pickleable and caching will work.

If convenient, you could build a simple pickleable class whose instances "stand for" queries, emulating whatever initialization and parameter-setting you require, and performing the just-in-time instantiation only when some method requiring an actual query object is called. But that's just a "convenience" idea, and does not alter the underlying concept as explained in the previous paragraph.

Answer 2

Try return list(results) instead of return results and see if it helps.

The beaker file cache needs to be able to pickle both cache keys and values; most iterators and generators are unpickleable.