Python 2.x gotcha's and landmines

Question

The purpose of my question is to strengthen my knowledge base with Python and get a better picture of it, which includes knowing its faults and surprises. To keep things specific, I'm only interested in the CPython interpreter.

I'm looking for something similar to what learned from my PHP landmines question where some of the answers were well known to me but a couple were borderline horrifying.

Update: Apparently one maybe two people are upset that I asked a question that's already partially answered outside of Stack Overflow. As some sort of compromise here's the URL http://www.ferg.org/projects/python_gotchas.html

Note that one or two answers here already are original from what was written on the site referenced above.

Answer 1

The only gotcha/surprise I've dealt with is with CPython's GIL. If for whatever reason you expect python threads in CPython to run concurrently... well they're not and this is pretty well documented by the Python crowd and even Guido himself.

A long but thorough explanation of CPython threading and some of the things going on under the hood and why true concurrency with CPython isn't possible. http://jessenoller.com/2009/02/01/python-threads-and-the-global-interpreter-lock/

Answer 2

Dynamic binding makes typos in your variable names surprisingly hard to find. It's easy to spend half an hour fixing a trivial bug.

EDIT: an example...

for item in some_list:
    ... # lots of code
... # more code
for tiem in some_other_list:
    process(item) # oops!

Answer 3

Expressions in default arguments are calculated when the function is defined, not when it’s called.

Example: consider defaulting an argument to the current time:

>>>import time
>>> def report(when=time.time()):
...     print when
...
>>> report()
1210294387.19
>>> time.sleep(5)
>>> report()
1210294387.19

The when argument doesn't change. It is evaluated when you define the function. It won't change until the application is re-started.

Strategy: you won't trip over this if you default arguments to None and then do something useful when you see it:

>>> def report(when=None):
...     if when is None:
...         when = time.time()
...     print when
...
>>> report()
1210294762.29
>>> time.sleep(5)
>>> report()
1210294772.23

Exercise: to make sure you've understood: why is this happening?

>>> def spam(eggs=[]):
...     eggs.append("spam")
...     return eggs
...
>>> spam()
['spam']
>>> spam()
['spam', 'spam']
>>> spam()
['spam', 'spam', 'spam']
>>> spam()
['spam', 'spam', 'spam', 'spam']

Answer 4

You should be aware of how class variables are handled in Python. Consider the following class hierarchy:

class AAA(object):
    x = 1

class BBB(AAA):
    pass

class CCC(AAA):
    pass

Now, check the output of the following code:

>>> print AAA.x, BBB.x, CCC.x
1 1 1
>>> BBB.x = 2
>>> print AAA.x, BBB.x, CCC.x
1 2 1
>>> AAA.x = 3
>>> print AAA.x, BBB.x, CCC.x
3 2 3

Surprised? You won't be if you remember that class variables are internally handled as dictionaries of a class object. If a variable name is not found in the dictionary of current class, the parent classes are searched for it. So, the following code again, but with explanations:

# AAA: {'x': 1}, BBB: {}, CCC: {}
>>> print AAA.x, BBB.x, CCC.x
1 1 1
>>> BBB.x = 2
# AAA: {'x': 1}, BBB: {'x': 2}, CCC: {}
>>> print AAA.x, BBB.x, CCC.x
1 2 1
>>> AAA.x = 3
# AAA: {'x': 3}, BBB: {'x': 2}, CCC: {}
>>> print AAA.x, BBB.x, CCC.x
3 2 3

Same goes for handling class variables in class instances (treat this example as a continuation of the one above):

>>> a = AAA()
# a: {}, AAA: {'x': 3}
>>> print a.x, AAA.x
3 3
>>> a.x = 4
# a: {'x': 4}, AAA: {'x': 3}
>>> print a.x, AAA.x
4 3

Answer 5

Loops and lambdas (or any closure, really): variables are bound by name

funcs = []
for x in range(5):
  funcs.append(lambda: x)

[f() for f in funcs]
# output:
# 5 5 5 5 5

A work around is either creating a separate function or passing the args by name:

funcs = []
for x in range(5):
  funcs.append(lambda x=x: x)
[f() for f in funcs]
# output:
# 1 2 3 4 5

Answer 6

There was a lot of discussion on hidden language features a while back: hidden-features-of-python. Where some pitfalls were mentioned (and some of the good stuff too).

Also you might want to check out Python Warts.

But for me, integer division's a gotcha:

>>> 5/2
2

You probably wanted:

>>> 5*1.0/2
2.5

If you really want this (C-like) behaviour, you should write:

>>> 5//2
2

As that will work with floats too (and it will work when you eventually go to Python 3):

>>> 5*1.0//2
2.0

GvR explains how integer division came to work how it does on the history of Python.

Answer 7

James Dumay eloquently reminded me of another Python gotcha:

Not all of Python's “included batteries” are wonderful.

James’ specific example was the HTTP libraries: httplib, urllib, urllib2, urlparse, mimetools, and ftplib. Some of the functionality is duplicated, and some of the functionality you'd expect is completely absent, e.g. redirect handling. Frankly, it's horrible.

If I ever have to grab something via HTTP these days, I use the urlgrabber module forked from the Yum project.

Answer 8

Floats are not printed at full precision by default (without repr):

x = 1.0 / 3
y = 0.333333333333
print x  #: 0.333333333333
print y  #: 0.333333333333
print x == y  #: False

repr prints too many digits:

print repr(x)  #: 0.33333333333333331
print repr(y)  #: 0.33333333333300003
print x == 0.3333333333333333  #: True

Answer 9

Not including an __init__.py in your packages. That one still gets me sometimes.

Answer 10

Unintentionally mixing oldstyle and newstyle classes can cause seemingly mysterious errors.

Say you have a simple class hierarchy consisting of superclass A and subclass B. When B is instantiated, A's constructor must be called first. The code below correctly does this:

class A(object):
    def __init__(self):
        self.a = 1

class B(A):
    def __init__(self):
        super(B, self).__init__()
        self.b = 1

b = B()

But if you forget to make A a newstyle class and define it like this:

class A:
    def __init__(self):
        self.a = 1

you get this traceback:

Traceback (most recent call last):
  File "AB.py", line 11, in <module>
    b = B()
  File "AB.py", line 7, in __init__
    super(B, self).__init__()
TypeError: super() argument 1 must be type, not classobj

Two other questions relating to this issue are 489269 and 770134

Answer 11

You cannot use locals()['x'] = whatever to change local variable values as you might expect.

This works:

>>> x = 1
>>> x
1
>>> locals()['x'] = 2
>>> x
2

BUT:

>>> def test():
...     x = 1
...     print x
...     locals()['x'] = 2
...     print x  # *** prints 1, not 2 ***
...
>>> test()
1
1

This actually burnt me in an answer here on SO, since I had tested it outside a function and got the change I wanted. Afterwards, I found it mentioned and contrasted to the case of globals() in "Dive Into Python." See example 8.12 in:

http://diveintopython.org/html_processing/locals_and_globals.html

(Though it does not note that the change via locals() will work at the top level as I show above.)