Iterating through large lists with potential conditions in Python

Question

I have large chunks of data, normally at around 2000+ entries, but in this report we have the ability to look as far as we want so it could be up to 10,000 records

The report is split up into: Two categories and then within each Category, we split by Currency so we have several sub categories within the list.

My issue comes in efficiently calculating the various subtotals. I am using Django and pass a templatetag the currency and category, if it applies, and then the templatetag renders the total. Note that sometimes I have a subtotal just for the category, with no currency passed.

Initially, I was using a seperate query for each subtotal by just using .filter() if there was a currency/category like so:

if currency:
  entries = entries.filter(item_currency=currency)

This became a problem as I would have too many queries, and too long of a generation time (2,000+ ms), so I opted to use list(entries) to execute my query right off the bat, and then loop through it with simple list comprehensions:

totals['quantity'] = sum([e.quantity for e in entries])

My problem if you don't see it yet, lies in .. how can I efficiently add the condition for currency / category on each list comprehension? Sometimes they won't be there, sometimes they will so I can't simply type:

totals['quantity'] = sum([e.quantity for e in entries if item_currency = currency])

I could make a huge if-block, but that's not very clean and is a maintenance disaster, so I'm reaching out to the Stackoverflow community for a bit of insight .. thanks in advance :)

Answer 1

You could define a little inline function:

def EntryMatches(e):
  if use_currency and not (e.currency == currency):
    return False
  if use_category and not (e.category == category):
    return False
  return True

then

totals['quantity'] = sum([e.quantity for e in entries if EntryMatches(e)])

EntryMatches() will have access to all variables in enclosing scope, so no need to pass in any more arguments. You get the advantage that all of the logic for which entries to use is in one place, you still get to use the list comprehension to make the sum() more readable, but you can have arbitrary logic in EntryMatches() now.