Why don't Python's list comprehensions make copies of arguments so actual objects can't be mutated?

Question

Maybe I've been drinking too much of the functional programming Kool Aid, but this behavior of list comprehensions seems like a bad design choice:

>>> d = [1, 2, 3, 4, 5]
>>> [d.pop() for _ in range(len(d))]
[5, 4, 3, 2, 1]
>>> d
[]

Why is d not copied, and then the copied lexically-scoped version not mutated (and then lost)? The point of list comprehensions seems like it should be to return the desired list, not return a list and silently mutate some other object behind the scenes. The destruction of d is somewhat implicit, which seems unPythonic. Is there a good use case for this?

Why is it advantageous to have list comps behave exactly like for loops, rather than behave more like functions (from a functional language, with local scope)?

Answer 1

Of course there is (e.g., queue processing). But clearly what you have shown isn't one.

Python, like any programming language worth using, does exactly what you tell it to, no more and no less. If you want it to do something else, then tell it to do something else.

Answer 2

d isn't copied because you didn't copy it and lists are mutable and pop contractually manipulates the list.

If you'd used a tuple, it would not have mutated:

>>> x = (1, 2, 3, 4)
>>> type(x)
<type 'tuple'>
>>> x.pop()
AttributeError: 'tuple' object has no attribute 'pop'

Answer 3

Python is not a functional language and never will be. Therefore, when you use a list comprehension, you can alter the state of unrelated data structures. This cannot be reasonably prevented, and measures such as the one you describe would only help the particular case you highlighted.

In general, it's up to the person using a list comprehension to write code that is fairly easy to understand and as free of side-effects as possible. I consider the code you posted to be bad programming style and a dumb way to create a reversed list when list.reverse exists.

Though, I suppose if the list you're popping from in that example is a queue that can be added to by the queue processing code (i.e. something much more complicated than d.pop()) or by another thread, then the code is sort of a reasonable way to do things. Though I really think it ought to be a loop and not a list comprehension.

Answer 4

Are you saying that you want methods to behave differently depending on the execution context? Sounds really dangerous to me.

It's a good thing that a method called on a Python object will always do the same thing - I'd be worried about using a language where calling a method inside some kind of syntactic construction caused it to behave differently.

Answer 5

There's always a way to not mutate the list when using list comprehensions. But you can mutate the list too, if that's what you want. In your case, for example:

c = [a for a in reversed(d)]
c = d[::-1]
c = [d[a] for a in xrange(len(d)-1, -1, -1)]

will all give you a reversed copy of the list. While

d.reverse()

will reverse the list in place.

Answer 6

You seem to be misunderstanding functions:

def fun(lst):
    for _ in range(len(lst)):
        lst.pop()

will have exactly the same effect as

(lst.pop() for _ in range(len(lst)))

This is because lst is not 'the' list but a reference to it. When you pass that reference around, it stays pointing to the same list. If you want the list copied, simply use lst[:]. If you want to copy its contents as well, use copy.deepcopy from the copy module.

Answer 7

In this expression:

[d.pop() for _ in range(len(d))]

what variable do you want to be implicitly copied or scoped? The only variable here with any kind of special status in the comprehension is _, which isn't the one you want protected.

I don't see how you could give list comprehensions semantics that could somehow identify all of the mutable variables involved, and somehow implicitly copy them. Or to know that .pop() changes its object?

You mention functional languages, but they accomplish what you want by making all variables immutable. Python simply isn't designed that way.

Answer 8

I'm not sure what you're asking. Maybe you're asking if d.pop() should return a copy instead of mutating itself. (That has nothing at all to do with list comprehensions.) The answer to that is no, of course not: that would turn it from an O(1) operation to O(n), which would be a catastrophic design flaw.

As for list comprehensions, nothing stops expressions within them from calling functions with side-effects. It's not the fault of list comprehensions if you misuse them. It's not the job of language design to forcefully prevent programmers from doing confusing things.

Answer 9

Why is it advantageous to have list comps behave exactly like for loops,

Because it's least surprising.

rather than behave more like functions (with local scope)?

What are you talking about? Functions can mutate their arguments:

>>> def mutate(d):
...     d.pop()
... 
>>> d = [1, 2, 3, 4, 5]
>>> mutate(d)
>>> d
[1, 2, 3, 4]

I see no inconsistency at all.

What you seem not to recognize is that Python is not a functional language. It's an imperative language that happens to have a few functional-like features. Python allows objects to be mutable. If you don't want them mutated, then just don't call methods like list.pop that are documented to mutate them.

Answer 10

Python never does copy unless you specifically ask it to do a copy. This is a perfectly simple, clear, and totally understandable rule. Putting exceptions and distinguos on it, such as "except under the following circumstances within a list comprehension...", would be utter folly: if Python's design had ever been under the management of somebody with such crazy ideas, Python would be a sick, contorted, half-broken language not worth learning. Thanks for making me happy all over again in the realization that is is definitely not the case!

You want a copy? Make a copy! That's always the solution in Python when you prefer a copy's overhead because you need to perform some changes that must not be reflected in the original. That is, in a clean approach, you'd do

dcopy = list(d)
[dcopy.pop() for _ in range(len(d))]

If you're super-keen to have everything within a single expression, you can, though it's possibly not code one would call exactly "clean":

[dcopy.pop() for dcopy in [list(d)] for _ in range(len(d))]

i.e., the usual trick one uses when one would really like to fold an assignment into a list comprehension (add a for clause, with the "control variable" being the name you want to assign to, and the "loop" is over a single-item sequence of the value you want to assign).

Functional languages never mutate data, therefore they don't make copies either (nor do they need to). Python is not a functional language, but of course there's a lot of things you can do in Python "the functional way", and often it's a better way. For example, a much better replacement for your list comprehension (guaranteed to have identical results and not affect d, and vastly faster, more concise, and cleaner):

d[::-1]

(AKA "the Martian Smiley", per my wife Anna;-). Slicing (not slice assignment, which is a different operation) always perform a copy in core Python (language and standard library), though of course not necessarily in independently developed third party modules like the popular numpy (which prefers to see a slice as a "view" on the original numpy.array).

Answer 11

Why should it create a (possibly very expensive) copy, when idiomatic code won't have side effects anyway? And why should the (rare, but existing) use cases where side effects are desired (and ok) be prohibited?

Python is first and foremost an imperative language. Mutable state is not only permitted, but essential - yeah, list comprehensions are intended to be pure, but if that was enforced, it would be asynch with the semantics of the rest of the language. So d.pop() mutates d, but only if it's not in a list comprehension and if the stars are right? That would be pointless. You're free (and supposed) not to make use of it, but nobody's going to set more rules in stone and make the feature more complex - idiomatic code (and that's the only code anyone should care about ;) ) doesn't need such a rule. It does so anyway, and does otherwise if needed.

Answer 12

Why is d not copied, and then the copied lexically-scoped version not mutated (and then lost)?

Because python is an object oriented programming language and doing so would be an incredibly bad idea. Everything is an object.

What makes you think it's possible to create "lexically scoped copies" of arbitrary objects?

Being able to call pop on an object doesn't mean it's possible to copy it. It might access a file handle, a network socket or an instruction queue for a space probe orbiting Saturn.

---

Why is it advantageous to have list comps behave exactly like for loops, rather than behave more like functions (with local scope)?

1. Because it creates concise, readable code.
2. As everyone else has pointed out, functions don't work in the way you appear to think they do. They don't do this "lexically scoped copies" thing either. I think you're getting confused with local assignment.

---

I recommend having a read of the articles here: http://www.cafepy.com/article/python_types_and_objects/python_types_and_objects.html

They are very informative about how python works.

Answer 13

dan04 has the right answer, and for comparison, here's a little Haskell...

[print s | s <- ["foo", "bar", "baz"]]

Here you have a side effect (printing) right in the middle of a list comprehension in Haskell. Haskell is lazy, so you have to explicitly run it with sequence_:

main = sequence_ [print s | s <- ["foo", "bar", "baz"]]

But that's practically the same as Python's

_ = list(print(s) for s in ["foo", "baz", "baz"])

Except that Haskell wraps the _ = list... idiom in function named sequence_.

List comprehensions don't have anything to do with preventing side effects. It's just unexpected to see them there. And you can hardly get more 'functional' than Haskell, so the answer "Python is an imperative language" isn't quite right in this context.