Most Important Python Idioms

Question

This question is inspired by several back-and-forths I've had recently about techniques that are or are not optimal for python programming. I may know other languages, but I am still learning in python. A wonderfully helpful comment was just recently posted by MikeD on a suboptimal answer of mine here that mentioned the operator module and Guido van Rossum's discussion of lisp-isms in python.

So, my question is what are the most common idioms that python programmers can be aware of so that their every-day code doesn't use well-documented-ly deprecated techniques that result in suboptimal code. I have a feeling many of these will deal with loop and iteration structures, or not blowing out memory by copying things unnecessarily, but some may not. I'm not talking about things that are common to all languages, like cacheing the results of an expression rather than calling the expression within a tight loop, I want to know just things for python. And not (necessarily) odd-ball things that one almost never runs into. I want my 80/20 effort maximized so I don't learn improper technique.

Most everyday-useful list wins the blue ribbon.

Update:

My original question used the word "optimizations" rather than idiom, so many of the answers assumed I was prematurely optimizing. Yes, writing idiomatic code is the correct answer, but what are your top, most important idioms?

Answer 1

Obligatory link: Python Speed/Performance tips.

Answer 2

I completely agree with SilentGhost's comment on writing idiomatic code.

If you compare Python to other languages (especially C), you'll discover that it is very much optimized for what I'll call the 'simple case' or the most straightforward solution. While in C you can optimize performance at the expense of creating unreadable code, in Python unreadable code is just that. I realize that this isn't as true today with smart compilers as it might have been years ago, but the principle is sound, I think.

As far as specific answers to your question, I'd think that Python programmers should be familiar with the String API, list comprehensions, maybe generators?

People sometimes ask me where I picked up a certain tidbit of information and it is hard to answer because there isn't one definitive source in the world. Just keep reading SO, and Google things that seem not quite right, and you'll eventually pick it all up.

Answer 3

The most important optimization in python is to not worry about optimization until you find your program is running less-than-optimal and there's a bottleneck.

And of course if you write idiomatic code as has been previously suggested, you'll rarely run into bottlenecks.

Answer 4

The Zen of Python (import this if you don't know it), while being only a collection of trivial wisdoms rather than a list of concrete idioms, is all-important. There's not an idiom for everything, but the Zen can be applied in almost every case. The remaining question is how exactly you apply it ;-)

Others may provide in-depth answer revolving around certain idioms. I, for my part, will just talk about one I use regularily, though too advanced to teach to every new Python user right away: Generator expressions.

Generator expressions are iterables (and thus lazy). Period. A list/dict/set comprehension is just a generator expression fed to the list/dict/set constructor. Omit the second step by just using a generator expression whenever you can - it will save hogs of memory and in possibly some time when processing much data. Example: Don't build a list of booleans via list comprehension (or for-loop which would make a great list comprehension) and feed it to all(). Just pass the underlying generator expression. If all values are True, you don't loose (much) performance, since you compute them all anyway. But if the first element of that 500 element list if False, you computed 499 elements too much. Plus, I feel warm and fuzzy about using a Haskell-like (lazy lists) feature in Python ;-)

But I wouldn't call e.g. the use of the operator module an idiom - it's just knowing the standard lib and choosing wisely from it. It's there for exactly that reason - of course it's more optimized than you own half-assed implementation, and of course any user of the language who knows it will naturally use it.

Admittedly, some might argue that this means there are no idioms, since it's all only about using the language's facilities wisely (taking advantage of lists being iterable instead of using range to get indexes, for example). I can't disagree, though I'd like to note that...

1. The operator module is not a part of the language but a library (which happens to be included in the standard library of the best-known implementation)
2. Its use is nowhere as universial as most idioms (most of the use cases are connected to higher-level iterators or functions - something few of us use very often in Python)
3. Not knowing that for actually iterates (not counts) is lightyears away from not remembering a specific module.

-> while certainly not the most pyhonic solution, it doesn't brand one as "total newbie who didn't even care to read the preface of a tutorial". Of course knowing the libraries is required to be a guru (in any language). But not knowing some isn't too bad imho.

Answer 5

Here you are three interesting links, IMHO:

First of all:

• Code Like a Pythonista: Idiomatic Python

and as a complement:

• Google Python Style Guide
• How not to write Python code

Answer 6

In addition to David Goodger's tutorial already mentioned by someone, in a similar category (perhaps in a broad sense):

• Rob Knight's Python Idioms and Efficiency
• Python Patterns - An Optimization Anecdote
• Alex Martelli's google talk on Python Design Patterns