Why doesn't this list comprehension do what I expect it to do?

Question

The original list project_keys = sorted(projects.keys()) is [101, 102, 103, 104, 105, 106, 107, 108, 109, 110] where the following projects were deemed invalid this year: 108, 109, 110.

Thus:

for project in projects.itervalues():
# The projects dictionary is mapped to the Project class
    if project.invalid:
    # Where invalid is a Bool parameter in the Project class
     project_keys.remove(project.proj_id)  

print project_keys

This will return a list of integers (which are project id's) as such:

[101, 102, 103, 104, 105, 106, 107]

Sweet.

Now, I wanted it try the same thing using a list comprehension.

project_keys = [project_keys.remove(project.proj_id) for project in projects.itervalues() if project.invalid  

print project_keys

This returns:

[None, None, None]

So I'm populating a list with the same number as the removed elements but they're Nones?

Can someone point out what I'm doing wrong?

Additionally, why would I use a list comprehension over the for-if block at the top? Conciseness? Looks nicer?

Answer 1

Your list comprehension works using side-effects. Just executing it should update project_keys to give the result you want.

[project_keys.remove(project.proj_id)
 for project in projects.itervalues()
 if project.invalid]

The return value from remove is None. Assigning the result of the list comprehension to project_keys is where you are going wrong.

A simple loop is probably clearer here though. A list comprehension that uses side-effects can be confusing.

However you can solve your problem in a slightly different way:

project_keys = sorted(project.proj_id
                      for project in projects.itervalues()
                      if not project.invalid)

This keeps the projects you are interested in, instead of removing those that you're not interested in. The example I gave above uses a generator expression instead of a list comprehension, but it would work with either.

Answer 2

You, sir, have misunderstood list comprehensions.

What you probably wanted (in words)

I want to remove all project ids that are invalid.

What you wrote

project_keys = [project_keys.remove(project.proj_id)
                for project in projects.itervalues() if project.invalid]

What is actually going on

dummy = []
for project in projects.itervalues():
  if project.invalid:
    dummy.append(project_keys.remove(project.proj_id)) #what are you
project_keys = dummy                                   #removing items from?
del dummy                                            

What is actually going on (now with more "functional")

mapped-fun = lambda project: project_keys.remove(project.proj_id)
filtering-fun = lambda project: project.invalid
project_keys = map(mapped-fun, filter(filtering-fun, projects.itervalues()))

---

As you can see, list comprehensions are not syntactical sugar around for loops. Rather, list comprehensions are syntactical sugar around map() and filter(): apply a function to all items in a sequence that match a condition and get a list of results in return.

Here, by function it is actually meant a side-effect-free transformation of input into output. This means that you "cannot" use methods that change the input itself, like list.sort(); you'll have to use their functional equivalents, like sorted().

By "cannot", however, I don't mean you'll get error messages or nasal demons; I mean you are abusing the language. In your case, the evaluation of the list comprehension that happens as you assign it to a variable does indeed produce the intended side-effects -- but does it produce them on the intended variables?

See, the only reason why this can execute without an error is that before this list comprehension, there was another list called project_keys and it's that list you are actually changing!

Lists comprehensions are a result of functional programming, which rejects side effects. Keep that in mind when using lists comprehensions.

---

So here's a thought process you can use to actually get the list comprehension you wanted.

What you actually wanted (in words)

I want all project ids that are valid (= not invalid.)

What you actually wanted

dummy = []
for project in projects.itervalues():
  if not project.invalid:
    dummy.append(project.proj_id)
project_keys = dummy
del dummy

What you actually wanted (now with more functional)

mapped-fun = lambda project: project.proj_id
filtering-fun = lambda project: not project.invalid
project_keys = map(mapped-fun, filter(filtering-fun, projects.itervalues()))

What you actually wanted (now as a list comprehension)

project_keys = [project.proj_id for project in projects.itervalues()
                if not project.invalid]