Class attribute evaluation and generators

Question

How exactly does Python evaluate class attributes? I've stumbled across an interesting quirk (in Python 2.5.2) that I'd like explained.

I have a class with some attributes that are defined in terms of other, previously defined attributes. When I try using a generator object, Python throws an error, but if I use a plain ordinary list comprehension, there's no problem.

Here's the pared-down example. Note that the only difference is that Brie uses a generator expression, while Cheddar uses a list comprehension.

# Using a generator expression as the argument to list() fails
>>> class Brie :
...     base = 2
...     powers = list(base**i for i in xrange(5))
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in Brie
  File "<stdin>", line 3, in <genexpr>
NameError: global name 'base' is not defined

# Using a list comprehension works
>>> class Cheddar :
...     base = 2
...     powers = [base**i for i in xrange(5)]
... 
>>> Cheddar.powers
[1, 2, 4, 8, 16]

# Using a list comprehension as the argument to list() works
>>> class Edam :
...     base = 2
...     powers = list([base**i for i in xrange(5)])
...
>>> Edam.powers
[1, 2, 4, 8, 16]

(My actual case was more complicated, and I was creating a dict, but this is the minimum example I could find.)

My only guess is that the list comprehensions are computed at that line, but the generator expressions are computed after the end of the class, at which point the scope has changed. But I'm not sure why the generator expression doesn't act as a closure and store the reference to base in the scope at the line.

Is there a reason for this, and if so, how should I be thinking of the evaluation mechanics of class attributes?

Answer 1

Yeah, it's a bit dodgy, this. A class doesn't really introduce a new scope, it just sort of looks a little bit like it does; constructs like this expose the difference.

The idea is that when you're using a generator expression it's equivalent to doing it with a lambda:

class Brie(object):
    base= 2
    powers= map(lambda i: base**i, xrange(5))

or explicitly as a function statement:

class Brie(object):
    base= 2

    def __generatePowers():
        for i in xrange(5):
            yield base**i

    powers= list(__generatePowers())

In this case it's clear that base isn't in scope for __generatePowers; an exception results for both (unless you were unlucky enough to also have a base global, in which case you get a wrongness).

This doesn't happen for list comprehensions due to some internal details on how they're evaluated, however that behaviour goes away in Python 3 which will fail equally for both cases. Some discussion here.

A workaround can be had using a lambda with the same technique we relied on back in the bad old days before nested_scopes:

class Brie(object):
    base= 2
    powers= map(lambda i, base= base: base**i, xrange(5))

Answer 2

From PEP 289:

After exploring many possibilities, a consensus emerged that binding issues were hard to understand and that users should be strongly encouraged to use generator expressions inside functions that consume their arguments immediately. For more complex applications, full generator definitions are always superior in terms of being obvious about scope, lifetime, and binding [6].

[6] (1, 2) Patch discussion and alternative patches on Source Forge http://www.python.org/sf/872326

It's how generator expressions are scoped as far as I can make out.