How do I create a list of Python lambdas (in a list comprehension/for loop)?

Question

I want to create a list of lambda objects from a list of constants in Python; for instance:

listOfNumbers = [1,2,3,4,5]
square = lambda x: x * x
listOfLambdas = [lambda: square(i) for i in listOfNumbers]

This will create a list of lambda objects, however, when I run them:

for f in listOfLambdas:
    print f(),

I would expect that it would print

1 4 9 16 25

Instead, it prints:

25 25 25 25 25

It seems as though the lambdas have all been given the wrong parameter. Have I done something wrong, and is there a way to fix it? I'm in Python 2.4 I think.

EDIT: a bit more of trying things and such came up with this:

listOfLambdas = []
for num in listOfNumbers:
    action = lambda: square(num)
    listOfLambdas.append(action)
    print action()

Prints the expected squares from 1 to 25, but then using the earlier print statement:

for f in listOfLambdas:
    print f(),

still gives me all 25s. How did the existing lambda objects change between those two print calls?

Related question: Why results of map() and list comprehension are different?

Answer 1

I'm guessing that the lambda you're creating in the list comprehension is bound to the variable i which eventually ends up at 5. Thus, when you evaluate the lambdas after the fact, they're all bound to 5 and end up calculating 25. The same thing is happening with num in your second example. When you evaluate the lambda inside the loop it's num hasn't changed so you get the right value. After the loop, num is 5...

I'm not quite sure what you're going for, so I'm not sure how to suggest a solution. How about this?

def square(x): return lambda : x*x
listOfLambdas = [square(i) for i in [1,2,3,4,5]]
for f in listOfLambdas: print f()

This gives me the expected output:

1
4
9
16
25

Another way to think of this is that a lambda "captures" its lexical environment at the point where it is created. So, if you give it num it doesn't actually resolve that value until its invoked. This is both confusing and powerful.

Answer 2

I sometimes find that defining actual classes for function objects makes it easier to understand what's going on:

>>> class square(object):
...   def __init__(self, val):
...     self.val = val
...   def __call__(self):
...     return self.val * self.val
...
>>> l = [1,2,3,4,5]
>>> funcs = [square(i) for i in l]
>>> for f in funcs:
...   print f()
...
1
4
9
16
25
>>>

Granted, it's a bit more verbose than using lambdas or closures, but I find this easier to understand when I'm trying to do non-obvious things with functions.

Answer 3

You have:

listOfLambdas = [lambda: i*i for i in range(6)]

for f in listOfLambdas:
    print f()

Output:

25
25
25
25
25
25

You need currying! Aside from being delicious, use this default value "hack".

listOfLambdas = [lambda i=i: i*i for i in range(6)]

for f in listOfLambdas:
    print f()

Output:

0
1
4
9
16
25

Note the i=i. That's where the magic happens. That and my rotating bed.

Answer 4

listOfLambdas = [lambda i=i: square(i) for i in listOfNumbers]

Or

listOfLambdas = map(lambda i: lambda: square(i), listOfNumbers)

Answer 5

When function statements are executed they are bound to their (lexically) enclosing scope.

In your snippet, the lambdas are bound to the global scope, because for suites are not executed as an independently scoped unit in Python. At the end of the for loop, the num is bound in the enclosing scope. Demo:

for num in range(1, 6):
    pass
assert num == 5 # num is now bound in the enclosing scope

So when you bind identifiers in the for loop you're actually manipulating the enclosing scope.

for num in range(1, 6):
    spam = 12
assert num == 5 # num is now bound in the enclosing scope
assert spam == 12 # spam is also bound in the enclosing scope

Same deal for list comprehensions:

[num for num in range(1, 6)]
assert num == 5

Mind blowing, I know. Anywho, with our newfound knowledge, we can determine that the lambdas you are creating are referring to the (single) num identifier bound in the enclosing scope. That should make this make more sense:

functions = []
for number in range(1, 6):
    def fun():
        return number
    functions.append(fun)
assert all(fun() == 5 for fun in functions)
assert all(fun() is number for fun in functions)

And here's the coolest part that demonstrates it even more:

# Same as above -- commented out for emphasis.
#functions = []
#for number in range(1, 6):
#    def fun():
#        return number
#    functions.append(fun)
#assert all(fun() == 5 for fun in functions)
#assert all(fun() is number for fun in functions)
number = 6 # Rebind 6 in the scope and see how it affects the results.
assert all(fun() == 6 for fun in functions)

So the solution to this, of course, is to make a new enclosing scope for each number you want to bind. In Python, you can create new enclosing scopes with modules, classes, and functions. It's common to use a function just to create new enclosing scope for another function.

In Python, a closure is a function that returns another function. Kind of like a function constructor. Check out get_fun in the following example:

def get_fun(value):
    """:return: A function that returns :param:`value`."""
    def fun(): # Bound to get_fun's scope
        return value
    return fun

functions = []
for number in range(1, 6):
    functions.append(get_fun(number))
assert [fun() for fun in functions] == range(1, 6)

Since get_fun is a function, it gets to have its own internal scope. Every time you call get_fun with a value, a little table is created to keep track of bindings within it; i.e. it says, "Within this scope, the value identifier points to the thing that was passed." That scope goes away at the end of the function execution, unless there's a reason for it to hang around.

If you're returning a function from within a scope, that's a good reason for parts of the "scope table" to hang around -- that function you're returning could reference things from that scope table when you call it later on. For that reason, when fun is created within get_fun Python tells fun about get_fun's scope table, which fun keeps handy for when it's needed.

You can read more about the details and technical terminology (which I softened a bit) in the Python docs on the execution model. You can also look at the parts of the enclosing scope that a function refers to with print fun.__closure__. In the above, we see the reference to the value, which happens to be an int:

# Same as before, commented out for emphasis.
#functions = []
#for number in range(1, 6):
#    functions.append(get_fun(number))
#assert [fun() for fun in functions] == range(1, 6)
print functions[0].__closure__
# Produces: (<cell at 0x8dc30: int object at 0x1004188>,)