Unexpected list comprehension behaviour in Python.

Question

I believe I'm getting bitten by some combination of nested scoping rules and list comprehensions. Jeremy Hylton's blog post is suggestive about the causes, but I don't really understand CPython's implementation well-enough to figure out how to get around this.

Here is an (overcomplicated?) example. If people have a simpler one that demos it, I'd like to hear it. The issue: the list comprehensions using next() are filled with the result from the last iteration.

edit: The Problem:

What exactly is going on with this, and how do I fix this? Do I have to use a standard for loop? Clearly the function is running the correct number of times, but the list comprehensions end up with the final value instead of the result of each loop.

Some hypotheses:

• generators?
• lazy filling of list comprehensions?

code

import itertools
def digit(n):
    digit_list = [ (x,False) for x in xrange(1,n+1)]
    digit_list[0] = (1,True)
    return itertools.cycle ( digit_list)

>>> D = digit(5)
>>> [D.next() for x in range(5)]
## This list comprehension works as expected
[(1, True), (2, False), (3, False), (4, False), (5, False)]

class counter(object):
    def __init__(self):
        self.counter = [ digit(4) for ii in range(2) ] 
        self.totalcount=0
        self.display = [0,] * 2
    def next(self):
        self.totalcount += 1
        self.display[-1] = self.counter[-1].next()[0]
        print self.totalcount, self.display
        return self.display

    def next2(self,*args):
        self._cycle(1)
        self.totalcount += 1
        print self.totalcount, self.display
        return self.display

    def _cycle(self,digit):
        d,first = self.counter[digit].next()
        #print digit, d, first
        #print self._display
        self.display[digit] = d
        if first and digit > 0:
            self._cycle(digit-1)


C = counter()
[C.next() for x in range(5)]
[C.next2() for x in range(5)]

OUTPUT

In [44]: [C.next() for x in range(6)]
1 [0, 1]
2 [0, 2]
3 [0, 3]
4 [0, 4]
5 [0, 1]
6 [0, 2]
Out[44]: [[0, 2], [0, 2], [0, 2], [0, 2], [0, 2], [0, 2]]

In [45]: [C.next2() for x in range(6)]
7 [0, 3]
8 [0, 4]
9 [1, 1]
10 [1, 2]
11 [1, 3]
12 [1, 4]
Out[45]: [[1, 4], [1, 4], [1, 4], [1, 4], [1, 4], [1, 4]]

# this should be:  [[0,3],[0,4]....[1,4]] or similar

Answer 1

The problem is that with return self.display you return a reference to this list (not a copy). So what you end up with is a list where each element is a reference to self.display. To illustrate, look at the following:

>>> a = [1,2]
>>> b = [a,a]
>>> b
[[1, 2], [1, 2]]
>>> a.append(3)
>>> b
[[1, 2, 3], [1, 2, 3]]

You probably want to use something like return self.display[:].

Answer 2

Mind if i refactor this a bit?

def digit(n):
    for i in itertools.count():
        yield (i%n+1, not i%n)

But actually you don't need that one, if you implement the whole thing as a simple iterator:

def counter(digits, base):
    counter = [0] * digits

    def iterator():
        for total in itertools.count(1):
            for i in range(len(counter)):
                counter[i] = (counter[i] + 1) % base
                if counter[i]:
                    break
            print total, list(reversed(counter))
            yield list(reversed(counter))

    return iterator()

c = counter(2, 4)
print list(itertools.islice(c, 10))

If you want to get rid of the print (debugging, is it?), go with a while-loop.

This incindentally also solves your initial problem, because reversed returns a copy of the list.

Oh, and it's zero-based now ;)