Caching values in Python list comprehensions

Question

I'm using the following list comprehension:

resources = [obj.get("file") for obj in iterator if obj.get("file") != None]

Is there a way to "cache" the value of obj.get("file") when it's checked in the if statement so that it doesn't have to call get again on obj when it generates the return list?

Answer 1

resources = filter(None, (obj.get("file") for obj in iterator))

See the documentation for filter for how to provide your own evaluation function. Passing None for the function (as above) filters out all values which aren't true.

If obj.get() returns an object which has a weird __nonzero__ method then you'd need to pass lambda obj: obj != None to get exactly the same result as your original code.

Answer 2

Try something like this:

resources = filter( lambda x: x is not None, [obj.get("file") for ob jin iterator])

Answer 3

Create a temporary dict to hold values. Then, create a function that uses this dict as a cache, and use that function in the list comprehension, like so:

obj_cache = {}

def cache_get (target, key):
    if (target, key) not in obj_cache: obj_cache[(target, key)] = target.get(key)
    return obj_cache[(target, key)]

resources = [cache_get(obj, "file") for obj in iterator if cache_get(obj, "file") != None]

Also, you probably already know this (and if so, please disregard this answer), but unless obj.get("file") is making a database call, opening a file, making a request over a network, or doing something else potentially expensive, calling it twice per iteration instead of once is probably harmless, since you're only adding O(n) to your cost.

Answer 4

If you want to stay with list / iterator comprehensions instead of using filter you can simply use:

resources = [file_obj
             for file_obj in (obj.get("file") for obj in iterator)
             if file_obj is not None]