Hidden features of Python

Question

What are the lesser-known but useful features of the Python programming language?

• Try to limit answers to Python core.
• One feature per answer.
• Give an example and short description of the feature, not just a link to documentation.
• Label the feature using a title as the first line.

Answer 1

Main messages :)

import this
# btw look at this module's source :)

---

De-cyphered:

The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than right now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Answer 2

List comprehensions

list comprehensions

Compare the more traditional (without list comprehension):

foo = []
for x in xrange(10):
  if x % 2 == 0:
     foo.append(x)

to:

foo = [x for x in xrange(10) if x % 2 == 0]

Answer 3

Metaclasses

of course :-) http://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python

Answer 4

Special methods

Absolute power!

Answer 5

Creating generators objects

If you write

x=(n for n in foo if bar(n))

you can get out the generator and assign it to x. Now it means you can do

for n in x:

The advantage of this is that you don't need intermediate storage, which you would need if you did

x = [n for n in foo if bar(n)]

In some cases this can lead to significant speed up.

You can append many if statements to the end of the generator, basically replicating nested for loops:

>>> n = ((a,b) for a in range(0,2) for b in range(4,6))
>>> for i in n:
...   print i 

(0, 4)
(0, 5)
(1, 4)
(1, 5)

Answer 6

Decorators

Decorators allow to wrap a function or method in another function that can add functionality, modify arguments or results, etc. You write decorators one line above the function definition, beginning with an "at" sign (@).

Example shows a print_args decorator that prints the decorated function's arguments before calling it:

>>> def print_args(function):
>>>     def wrapper(*args, **kwargs):
>>>         print 'Arguments:', args, kwargs
>>>         return function(*args, **kwargs)
>>>     return wrapper

>>> @print_args
>>> def write(text):
>>>     print text

>>> write('foo')
Arguments: ('foo',) {}
foo

Answer 7

Readable regular expressions

In Python you can split a regular expression over multiple lines, name your matches and insert comments.

Example verbose syntax (from Dive into Python):

>>> pattern = """
... ^                   # beginning of string
... M{0,4}              # thousands - 0 to 4 M's
... (CM|CD|D?C{0,3})    # hundreds - 900 (CM), 400 (CD), 0-300 (0 to 3 C's),
...                     #            or 500-800 (D, followed by 0 to 3 C's)
... (XC|XL|L?X{0,3})    # tens - 90 (XC), 40 (XL), 0-30 (0 to 3 X's),
...                     #        or 50-80 (L, followed by 0 to 3 X's)
... (IX|IV|V?I{0,3})    # ones - 9 (IX), 4 (IV), 0-3 (0 to 3 I's),
...                     #        or 5-8 (V, followed by 0 to 3 I's)
... $                   # end of string
... """
>>> re.search(pattern, 'M', re.VERBOSE)

Example naming matches (from Regular Expression HOWTO)

>>> p = re.compile(r'(?P<word>\b\w+\b)')
>>> m = p.search( '(((( Lots of punctuation )))' )
>>> m.group('word')
'Lots'

You can also verbosely write a regex without using re.VERBOSE thanks to string literal concatenation.

>>> pattern = (
...     "^"                 # beginning of string
...     "M{0,4}"            # thousands - 0 to 4 M's
...     "(CM|CD|D?C{0,3})"  # hundreds - 900 (CM), 400 (CD), 0-300 (0 to 3 C's),
...                         #            or 500-800 (D, followed by 0 to 3 C's)
...     "(XC|XL|L?X{0,3})"  # tens - 90 (XC), 40 (XL), 0-30 (0 to 3 X's),
...                         #        or 50-80 (L, followed by 0 to 3 X's)
...     "(IX|IV|V?I{0,3})"  # ones - 9 (IX), 4 (IV), 0-3 (0 to 3 I's),
...                         #        or 5-8 (V, followed by 0 to 3 I's)
...     "$"                 # end of string
... )
>>> print pattern
"^M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$"

Answer 8

Nested list comprehensions and generator expressions:

[(i,j) for i in range(3) for j in range(i) ]    
((i,j) for i in range(4) for j in range(i) )

These can replace huge chunks of nested-loop code.

Answer 9

Getter functions in module operator

The functions attrgetter() and itemgetter() in module operator can be used to generate fast access functions for use in sorting and search objects and dictionaries

Chapter 6.7 in the Python Library Docs

Answer 10

Sending values into generator functions. For example having this function:

def mygen():
  """Yield 5 until something else is passed back via send()"""
  a = 5
  while True:
    f = yield(a) #yield a and possibly get f in return
    if f is not None: a = f  #store the new value

You can:

>>> g = mygen()
>>> g.next()
5
>>> g.next()
5
>>> g.send(7)  #we send this back to the generator
7
>>> g.next() #now it will yield 7 until we send something else
7

Answer 11

Ability to substitute even things like file deletion, file opening etc. - direct manipulation of language library. This is a huge advantage when testing. You don't have to wrap everything in complicated containers. Just substitute a function/method and go. This is also called monkey-patching.

Answer 12

>>> x=[1,1,2,'a','a',3]
>>> y = [ _x for _x in x if not _x in locals()['_[1]'] ]
>>> y
[1, 2, 'a', 3]

"locals()['_[1]']" is the "secret name" of the list being created. Very useful when state of list being built affects subsequent build decisions.

Answer 13

The step argument in slice operators. For example:

a = [1,2,3,4,5]
>>> a[::2]  # iterate over the whole list in 2-increments
[1,3,5]

The special case x[::-1] is a useful idiom for 'x reversed'.

>>> a[::-1]
[5,4,3,2,1]

Answer 14

Implicit concatenation:

>>> print "Hello " "World"
Hello World

Useful when you want to make a long text fit on several lines in a script:

hello = "Greaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa Hello " \
        "Word"

or

hello = ("Greaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa Hello " 
         "Word")

Answer 15

property

class ClassName(object):
    """
    """    
    def __init__(self, foo, bar):
        """
        """
        self.foo = foo # read-write property
        self.bar = bar # simple attribute

    def _set_foo(self, value):
        self._foo = value

    def _get_foo(self):
        return self._foo

    foo = property(_get_foo, _set_foo)

In Python 2.6 and 3.0:

class C(object):
    @property
    def x(self):
        return self._x

    @x.setter
    def x(self, value):
        self._x = value

    @x.deleter
    def x(self):
        del self._x

class D(C):
    @C.x.getter
    def x(self):
        return self._x * 2

    @x.setter
    def x(self, value):
        self._x = value / 2

Answer 16

Chaining comparison operators:

>>> x = 5
>>> 1 < x < 10
True
>>> 10 < x < 20 
False
>>> x < 10 < x*10 < 100
True
>>> 10 > x <= 9
True
>>> 5 == x > 4
True

In case you're thinking it's doing 1 < x, which comes out as True, and then comparing True < 10, which is also True, then no, that's really not what happens (see the last example.) It's really translating into 1 < x and x < 10, and x < 10 and 10 < x * 10 and x*10 < 100, but with less typing and each term is only evaluated once.

Answer 17

Everything is dynamic

"There is no compile-time". Everything in Python is runtime. A module is 'defined' by executing the module's source top-to-bottom, just like a script, and the resulting namespace is the module's attribute-space. Likewise, a class is 'defined' by executing the class body top-to-bottom, and the resulting namespace is the class's attribute-space. A class body can contain completely arbitrary code -- including import statements, loops and other class statements. Creating a class, function or even module 'dynamically', as is sometimes asked for, isn't hard; in fact, it's impossible to avoid, since everything is 'dynamic'.

Answer 18

Re-raising exceptions:

# Python 2 syntax
try:
    some_operation()
except SomeError, e:
    if is_fatal(e):
        raise
    handle_nonfatal(e)

# Python 3 syntax
try:
    some_operation()
except SomeError as e:
    if is_fatal(e):
        raise
    handle_nonfatal(e)

The 'raise' statement with no arguments inside an error handler tells Python to re-raise the exception with the original traceback intact, allowing you to say "oh, sorry, sorry, I didn't mean to catch that, sorry, sorry."

If you wish to print, store or fiddle with the original traceback, you can get it with sys.exc_info(), and printing it like Python would is done with the 'traceback' module.

Answer 19

In-place value swapping

>>> a = 10
>>> b = 5
>>> a, b
(10, 5)

>>> a, b = b, a
>>> a, b
(5, 10)

The right-hand side of the assignment is an expression that creates a new tuple. The left-hand side of the assignment immediately unpacks that (unreferenced) tuple to the names a and b.

After the assignment, the new tuple is unreferenced and marked for garbage collection, and the values bound to a and b have been swapped.

As noted in the Python tutorial section on data structures,

Note that multiple assignment is really just a combination of tuple packing and sequence unpacking.

Answer 20

Descriptors

They're the magic behind a whole bunch of core Python features.

When you use dotted access to look up a member (eg, x.y), Python first looks for the member in the instance dictionary. If it's not found, it looks for it in the class dictionary. If it finds it in the class dictionary, and the object implements the descriptor protocol, instead of just returning it, Python executes it. A descriptor is any class that implements the __get__, __set__, or __del__ methods.

Here's how you'd implement your own (read-only) version of property using descriptors:

class Property(object):
    def __init__(self, fget):
        self.fget = fget

    def __get__(self, obj, type):
        if obj is None:
            return self
        return self.fget(obj)

and you'd use it just like the built-in property():

class MyClass(object):
    @Property
    def foo(self):
        return "Foo!"

Descriptors are used in Python to implement properties, bound methods, static methods, class methods and slots, amongst other things. Understanding them makes it easy to see why a lot of things that previously looked like Python 'quirks' are the way they are.

Raymond Hettinger has an excellent tutorial that does a much better job of describing them than I do.

Answer 21

Doctest: documentation and unit-testing at the same time.

Example extracted from the Python documentation:

def factorial(n):
    """Return the factorial of n, an exact integer >= 0.

    If the result is small enough to fit in an int, return an int.
    Else return a long.

    >>> [factorial(n) for n in range(6)]
    [1, 1, 2, 6, 24, 120]
    >>> factorial(-1)
    Traceback (most recent call last):
        ...
    ValueError: n must be >= 0

    Factorials of floats are OK, but the float must be an exact integer:
    """

    import math
    if not n >= 0:
        raise ValueError("n must be >= 0")
    if math.floor(n) != n:
        raise ValueError("n must be exact integer")
    if n+1 == n:  # catch a value like 1e300
        raise OverflowError("n too large")
    result = 1
    factor = 2
    while factor <= n:
        result *= factor
        factor += 1
    return result

def _test():
    import doctest
    doctest.testmod()    

if __name__ == "__main__":
    _test()

Answer 22

iter() can take a callable argument

For instance:

def seek_next_line(f):
    for c in iter(lambda: f.read(1),'\n'):
        pass

The iter(callable, until_value) function repeatedly calls callable and yields its result until until_value is returned.

Answer 23

Too lazy to initialize every field in a dictionary? No problem:

In Python > 2.3:

from collections import defaultdict

In Python <= 2.3:

def defaultdict(type_):
    class Dict(dict):
        def __getitem__(self, key):
            return self.setdefault(key, type_())
    return Dict()

In any version:

d = defaultdict(list)
for stuff in lots_of_stuff:
     d[stuff.name].append(stuff)

Answer 24

The Python Interpreter

>>>

Maybe not lesser known, but certainly one of my favorite features of Python.

Answer 25

First-class functions

It's not really a hidden feature, but the fact that functions are first class objects is simply great. You can pass them around like any other variable.

>>> def jim(phrase):
...   return 'Jim says, "%s".' % phrase
>>> def say_something(person, phrase):
...   print person(phrase)

>>> say_something(jim, 'hey guys')
'Jim says, "hey guys".'

Answer 26

Creating new types at runtime

>>> NewType = type("NewType", (object,), {"x": "hello"})
>>> n = NewType()
>>> n.x
"hello"

which is exactly the same as

>>> class NewType(object):
>>>     x = "hello"
>>> n = NewType()
>>> n.x
"hello"

Probably not the most useful thing, but nice to know.

Edit: Fixed name of new type, should be NewType to be the exact same thing as with class statement.

Answer 27

Interleaving if and for in list comprehensions

>>> [(x, y) for x in range(4) if x % 2 == 1 for y in range(4)]
[(1, 0), (1, 1), (1, 2), (1, 3), (3, 0), (3, 1), (3, 2), (3, 3)]

I never realized this until I learned Haskell.

Answer 28

Context managers and the "with" Statement

Introduced in PEP 343, a context manager is an object that acts as a run-time context for a suite of statements.

Since the feature makes use of new keywords, it is introduced gradually: it is available in Python 2.5 via the __future__ directive. Python 2.6 and above (including Python 3) has it available by default.

I have used the "with" statement a lot because I think it's a very useful construct, here is a quick demo:

from __future__ import with_statement

with open('foo.txt', 'w') as f:
    f.write('hello!')

What's happening here behind the scenes, is that the "with" statement calls the special __enter__ and __exit__ methods on the file object. Exception details are also passed to __exit__ if any exception was raised from the with statement body, allowing for exception handling to happen there.

What this does for you in this particular case is that it guarantees that the file is closed when execution falls out of scope of the with suite, regardless if that occurs normally or whether an exception was thrown. It is basically a way of abstracting away common exception-handling code.

Other common use cases for this include locking with threads and database transactions.

Answer 29

Some of the builtin favorites, map(), reduce(), and filter(). All extremely fast and powerful.

Answer 30

Function argument unpacking

You can unpack a list or a dictionary as function arguments using * and **.

For example:

def draw_point(x, y):
    # do some magic

point_foo = (3, 4)
point_bar = {'y': 3, 'x': 2}

draw_point(*point_foo)
draw_point(**point_bar)

Very useful shortcut since lists, tuples and dicts are widely used as containers.

Answer 31

Dictionaries have a 'get()' method. If you do d['key'] and key isn't there, you get an exception. If you do d.get('key'), you get back None if 'key' isn't there. You can add a second argument to get that item back instead of None, eg: d.get('key', 0).

It's great for things like adding up numbers:

sum[value] = sum.get(value, 0) + 1

Answer 32

If you use exec in a function the variable lookup rules change drastically. Closures are no longer possible but Python allows arbitrary identifiers in the function. This gives you a "modifiable locals()" and can be used to star-import identifiers. On the downside it makes every lookup slower because the variables end up in a dict rather than slots in the frame:

>>> def f():
...  exec "a = 42"
...  return a
... 
>>> def g():
...  a = 42
...  return a
... 
>>> import dis
>>> dis.dis(f)
  2           0 LOAD_CONST               1 ('a = 42')
              3 LOAD_CONST               0 (None)
              6 DUP_TOP             
              7 EXEC_STMT           

  3           8 LOAD_NAME                0 (a)
             11 RETURN_VALUE        
>>> dis.dis(g)
  2           0 LOAD_CONST               1 (42)
              3 STORE_FAST               0 (a)

  3           6 LOAD_FAST                0 (a)
              9 RETURN_VALUE

Answer 33

From 2.5 onwards dicts have a special method __missing__ that is invoked for missing items:

>>> class MyDict(dict):
...  def __missing__(self, key):
...   self[key] = rv = []
...   return rv
... 
>>> m = MyDict()
>>> m["foo"].append(1)
>>> m["foo"].append(2)
>>> dict(m)
{'foo': [1, 2]}

There is also a dict subclass in collections called defaultdict that does pretty much the same but calls a function without arguments for not existing items:

>>> from collections import defaultdict
>>> m = defaultdict(list)
>>> m["foo"].append(1)
>>> m["foo"].append(2)
>>> dict(m)
{'foo': [1, 2]}

I recommend converting such dicts to regular dicts before passing them to functions that don't expect such subclasses. A lot of code uses d[a_key] and catches KeyErrors to check if an item exists which would add a new item to the dict.

Answer 34

If you are using descriptors on your classes Python completely bypasses __dict__ for that key which makes it a nice place to store such values:

>>> class User(object):
...  def _get_username(self):
...   return self.__dict__['username']
...  def _set_username(self, value):
...   print 'username set'
...   self.__dict__['username'] = value
...  username = property(_get_username, _set_username)
...  del _get_username, _set_username
... 
>>> u = User()
>>> u.username = "foo"
username set
>>> u.__dict__
{'username': 'foo'}

This helps to keep dir() clean.

Answer 35

If you don't like using whitespace to denote scopes, you can use the C-style {} by issuing:

from __future__ import braces

Answer 36

__slots__ is a nice way to save memory, but it's very hard to get a dict of the values of the object. Imagine the following object:

class Point(object):
    __slots__ = ('x', 'y')

Now that object obviously has two attributes. Now we can create an instance of it and build a dict of it this way:

>>> p = Point()
>>> p.x = 3
>>> p.y = 5
>>> dict((k, getattr(p, k)) for k in p.__slots__)
{'y': 5, 'x': 3}

This however won't work if point was subclassed and new slots were added. However Python automatically implements __reduce_ex__ to help the copy module. This can be abused to get a dict of values:

>>> p.__reduce_ex__(2)[2][1]
{'y': 5, 'x': 3}

Answer 37

Python's advanced slicing operation has a barely known syntax element, the ellipsis:

>>> class C(object):
...  def __getitem__(self, item):
...   return item
... 
>>> C()[1:2, ..., 3]
(slice(1, 2, None), Ellipsis, 3)

Unfortunately it's barely useful as the ellipsis is only supported if tuples are involved.

Answer 38

Builtin methods or functions don't implement the descriptor protocol which makes it impossible to do stuff like this:

>>> class C(object):
...  id = id
... 
>>> C().id()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: id() takes exactly one argument (0 given)

However you can create a small bind descriptor that makes this possible:

>>> from types import MethodType
>>> class bind(object):
...  def __init__(self, callable):
...   self.callable = callable
...  def __get__(self, obj, type=None):
...   if obj is None:
...    return self
...   return MethodType(self.callable, obj, type)
... 
>>> class C(object):
...  id = bind(id)
... 
>>> C().id()
7414064

Answer 39

Named formatting, % -formatting takes a dictionary (also applies %i/%s etc. validation).

>>> print "The %(foo)s is %(bar)i." % {'foo': 'answer', 'bar':42}
The answer is 42.

>>> foo, bar = 'question', 123

>>> print "The %(foo)s is %(bar)i." % locals()
The question is 123.

And since locals() is also a dictionary, you can simply pass that as a dict and have % -substitions from your local variables. I think this is frowned upon, but simplifies things..

New Style Formatting

>>> print "The {foo} is {bar}".format(foo='answer', bar=42)

Answer 40

Be careful with mutable default arguments

>>> def foo(x=[]):
...     x.append(1)
...     print x
... 
>>> foo()
[1]
>>> foo()
[1, 1]
>>> foo()
[1, 1, 1]

Instead, you should use a sentinel value denoting "not given" and replace with the mutable you'd like as default:

>>> def foo(x=None):
...     if x is None:
...         x = []
...     x.append(1)
...     print x
>>> foo()
[1]
>>> foo()
[1]

Answer 41

Tuple unpacking:

>>> (a, (b, c), d) = [(1, 2), (3, 4), (5, 6)]
>>> a
(1, 2)
>>> b
3
>>> c, d
(4, (5, 6))

More obscurely, you can do this in function arguments (in Python 2.x; Python 3.x will not allow this anymore):

>>> def addpoints((x1, y1), (x2, y2)):
...     return (x1+x2, y1+y2)
>>> addpoints((5, 0), (3, 5))
(8, 5)

Answer 42

unzip un-needed in Python

Someone blogged about Python not having an unzip function to go with zip(). unzip is straight-forward to calculate because:

>>> t1 = (0,1,2,3)
>>> t2 = (7,6,5,4)
>>> [t1,t2] == zip(*zip(t1,t2))
True

On reflection though, I'd rather have an explicit unzip().

Answer 43

To add more python modules (espcially 3rd party ones), most people seem to use PYTHONPATH environment variables or they add symlinks or directories in their site-packages directories. Another way, is to use *.pth files. Here's the official python doc's explanation:

"The most convenient way [to modify python's search path] is to add a path configuration file to a directory that's already on Python's path, usually to the .../site-packages/ directory. Path configuration files have an extension of .pth, and each line must contain a single path that will be appended to sys.path. (Because the new paths are appended to sys.path, modules in the added directories will not override standard modules. This means you can't use this mechanism for installing fixed versions of standard modules.)"

Answer 44

Exception else clause:

try:
  put_4000000000_volts_through_it(parrot)
except Voom:
  print "'E's pining!"
else:
  print "This parrot is no more!"
finally:
  end_sketch()

The use of the else clause is better than adding additional code to the try clause because it avoids accidentally catching an exception that wasn’t raised by the code being protected by the try ... except statement.

See http://docs.python.org/tut/node10.html

Answer 45

The for...else idiom (see http://docs.python.org/ref/for.html )

for i in foo:
    if i == 0:
        break
else:
    print("i was never 0")

The "else" block will be normally executed at the end of the for loop, unless the break is called.

The above code could be emulated as follows:

found = False
for i in foo:
    if i == 0:
        found = True
        break
if not found: 
    print("i was never 0")

Answer 46

Many people don't know about the "dir" function. It's a great way to figure out what an object can do from the interpreter. For example, if you want to see a list of all the string methods:

>>> dir("foo")
['__add__', '__class__', '__contains__', (snipped a bunch), 'title',
 'translate', 'upper', 'zfill']

And then if you want more information about a particular method you can call "help" on it.

>>> help("foo".upper)
    Help on built-in function upper:

upper(...)
    S.upper() -> string

    Return a copy of the string S converted to uppercase.

Answer 47

class AttrDict(dict):

    def __getattr__(self, name):
        if name in self:
            return self[name]
        raise AttributeError('%s not found' % name)

    def __setattr__(self, name, value):
        self[name] = value

    def __delattr__(self, name):
        del self[name]

person = AttrDict({'name': 'John Doe', 'age': 66})
print person['name']
print person.name

person.name = 'Frodo G'
print person.name

del person.age

print person

Answer 48

Python sort function sorts tuples correctly (i.e. using the familiar lexicographical order):

a = [(2, "b"), (1, "a"), (2, "a"), (3, "c")]
print sorted(a)
#[(1, 'a'), (2, 'a'), (2, 'b'), (3, 'c')]

Useful if you want to sort a list of persons after age and then name.

Answer 49

Conditional Assignment

x = 3 if (y == 1) else 2

It does exactly what it sounds like: "assign 3 to x if y is 1, otherwise assign 2 to x". Note that the parens are not necessary, but I like them for readability. You can also chain it if you have something more complicated:

x = 3 if (y == 1) else 2 if (y == -1) else 1

Though at a certain point, it goes a little too far.

Answer 50

• The underscore, it contains the most recent output value displayed by the interpreter (in an interactive session):

>>> (a for a in xrange(10000))
<generator object at 0x81a8fcc>
>>> b = 'blah'
>>> _
<generator object at 0x81a8fcc>

• A convenient Web-browser controller:

>>> import webbrowser
>>> webbrowser.open_new_tab('http://www.stackoverflow.com')

• A built-in http server. To serve the files in the current directory:

python -m SimpleHTTPServer 8000

• AtExit

>>> import atexit

Answer 51

__getattr__()

getattr is a really nice way to make generic classes, which is especially useful if you're writing an API. For example, in the FogBugz Python API, getattr is used to pass method calls on to the web service seamlessly:

class FogBugz:
    ...

    def __getattr__(self, name):
        # Let's leave the private stuff to Python
        if name.startswith("__"):
            raise AttributeError("No such attribute '%s'" % name)

        if not self.__handlerCache.has_key(name):
            def handler(**kwargs):
                return self.__makerequest(name, **kwargs)
            self.__handlerCache[name] = handler
        return self.__handlerCache[name]
    ...

When someone calls FogBugz.search(q='bug'), they don't get actually call a search method. Instead, getattr handles the call by creating a new function that wraps the makerequest method, which crafts the appropriate HTTP request to the web API. Any errors will be dispatched by the web service and passed back to the user.

Answer 52

enumerate

Wrap an iterable with enumerate and it will yield the item along with its index.

For example:

>>> a = ['a', 'b', 'c', 'd', 'e']
>>> for index, item in enumerate(a): print index, item
...
0 a
1 b
2 c
3 d
4 e
>>>

References:

• Python tutorial—looping techniques
• Python docs—built-in functions—enumerate
• PEP 279

Answer 53

Ternary operator

>>> 'ham' if True else 'spam'
'ham'
>>> 'ham' if False else 'spam'
'spam'

This was added in 2.5, prior to that you could use:

>>> True and 'ham' or 'spam'
'ham'
>>> False and 'ham' or 'spam'
'spam'

However, if the values you want to work with would be considered false, there is a difference:

>>> [] if True else 'spam'
[]
>>> True and [] or 'spam'
'spam'

Answer 54

You can build up a dictionary from a set of length-2 sequences. Extremely handy when you have a list of values and a list of arrays.

>>> dict([ ('foo','bar'),('a',1),('b',2) ])
{'a': 1, 'b': 2, 'foo': 'bar'}

>>> names = ['Bob', 'Marie', 'Alice']
>>> ages = [23, 27, 36]
>>> dict(zip(names, ages))
{'Alice': 36, 'Bob': 23, 'Marie': 27}

Answer 55

Tuple unpacking in for loops, list comprehensions and generator expressions:

>>> l=[(1,2),(3,4)]
>>> [a+b for a,b in l ] 
[3,7]

Useful in this idiom for iterating over (key,data) pairs in dictionaries:

d = { 'x':'y', 'f':'e'}
for name, value in d.items():  # one can also use iteritems()
   print "name:%s, value:%s" % (name,value)

prints:

name:x, value:y
name:f, value:e

Answer 56

"Unpacking" to function parameters

def foo(a, b, c):
        print a, b, c

bar = (3, 14, 15)
foo(*bar)

When executed prints:

3 14 15

Answer 57

Objects in boolean context

Empty tuples, lists, dicts, strings and many other objects are equivalent to False in boolean context (and non-empty are equivalent to True).

empty_tuple = ()
empty_list = []
empty_dict = {}
empty_string = ''
empty_set = set()
if empty_tuple or empty_list or empty_dict or empty_string or empty_set:
  print 'Never happens!'

This allows logical operations to return one of it's operands instead of True/False, which is useful in some situations:

s = t or "Default value" # s will be assigned "Default value"
                         # if t is false/empty/none

Answer 58

The first-classness of everything ('everything is an object'), and the mayhem this can cause.

>>> x = 5
>>> y = 10
>>> 
>>> def sq(x):
...   return x * x
... 
>>> def plus(x):
...   return x + x
... 
>>> (sq,plus)[y>x](y)
20

The last line creates a tuple containing the two functions, then evaluates y>x (True) and uses that as an index to the tuple (by casting it to an int, 1), and then calls that function with parameter y and shows the result.

For further abuse, if you were returning an object with an index (e.g. a list) you could add further square brackets on the end; if the contents were callable, more parentheses, and so on. For extra perversion, use the result of code like this as the expression in another example (i.e. replace y>x with this code):

(sq,plus)[y>x](y)[4](x)

This showcases two facets of Python - the 'everything is an object' philosophy taken to the extreme, and the methods by which improper or poorly-conceived use of the language's syntax can lead to completely unreadable, unmaintainable spaghetti code that fits in a single expression.

Answer 59

Assigning and deleting slices:

>>> a = range(10)
>>> a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> a[:5] = [42]
>>> a
[42, 5, 6, 7, 8, 9]
>>> a[:1] = range(5)
>>> a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> del a[::2]
>>> a
[1, 3, 5, 7, 9]
>>> a[::2] = a[::-2]
>>> a
[9, 3, 5, 7, 1]

Note: when assigning to extended slices (s[start:stop:step]), the assigned iterable must have the same length as the slice.

Answer 60

dict's constructor accepts keyword arguments:

>>> dict(foo=1, bar=2)
{'foo': 1, 'bar': 2}

Answer 61

Not an out-of-the-box feature, but Pyrex is incredibly useful.

Answer 62

Get the python regex parse tree to debug your regex

Regular expression are a great feature of python, but debugging them can be a pain, and it's just too easy to get a regex wrong.

Fortunately, python have a really hidden feature to print the regex parse tree, by passing the undocumented, experimental, hidden flag re.DEBUG (actually, 128) to re.compile

>>> re.compile("^\[font(?:=(?P<size>[-+][0-9]{1,2}))?\](.*?)[/font]",
    re.DEBUG)
at at_beginning
literal 91
literal 102
literal 111
literal 110
literal 116
max_repeat 0 1
  subpattern None
    literal 61
    subpattern 1
      in
        literal 45
        literal 43
      max_repeat 1 2
        in
          range (48, 57)
literal 93
subpattern 2
  min_repeat 0 65535
    any None
in
  literal 47
  literal 102
  literal 111
  literal 110
  literal 116

Once you understand the syntax, you can spot your errors. There we can see that i forgot to escape the [] in [/font].

Of course you can combine it with whatever flags you want, like commented regexes :

>>> re.compile("""
 ^              # start of a line
 \[font         # the font tag
 (?:=(?P<size>  # optional [font=+size]
 [-+][0-9]{1,2} # size specification
 ))?
 \]             # end of tag
 (.*?)          # text beetween the tags
 \[/font\]      # end of the tag
 """, re.DEBUG|re.VERBOSE|re.DOTALL)

Answer 63

Built-in base64, zlib, and rot13 codecs

Strings have encode and decode methods. Usually this is used for converting str to unicode and vice versa, e.g. with u = s.encode('utf8'). But there are some other handy builtin codecs. Compression and decompression with zlib (and bz2) is available without an explicit import:

>>> s = 'a' * 100
>>> s.encode('zlib')
'x\x9cKL\xa4=\x00\x00zG%\xe5'

Similarly you can encode and decode base64:

>>> 'Hello world'.encode('base64')
'SGVsbG8gd29ybGQ=\n'
>>> 'SGVsbG8gd29ybGQ=\n'.decode('base64')
'Hello world'

And, of course, you can rot13:

>>> 'Secret message'.encode('rot13')
'Frperg zrffntr'

Answer 64

Obviously, the antigravity module. xkcd #353

Answer 65

Generators

I think that a lot of beginning Python developers pass over generators without really grasping what they're for or getting any sense of their power. It wasn't until I read David M. Beazley's PyCon presentation on generators (it's available here) that I realized how useful (essential, really) they are. That presentation illuminated what was for me an entirely new way of programming, and I recommend it to anyone who doesn't have a deep understanding of generators.

Answer 66

Interactive Interpreter Tab Completion

try:
    import readline
except ImportError:
    print "Unable to load readline module."
else:
    import rlcompleter
    readline.parse_and_bind("tab: complete")


>>> class myclass:
...    def function(self):
...       print "my function"
... 
>>> class_instance = myclass()
>>> class_instance.<TAB>
class_instance.__class__   class_instance.__module__
class_instance.__doc__     class_instance.function
>>> class_instance.f<TAB>unction()

You will also have to set a PYTHONSTARTUP environment variable.

Answer 67

Python has GOTO

...and it's implemented by external pure-Python module :)

from goto import goto, label
for i in range(1, 10):
    for j in range(1, 20):
        for k in range(1, 30):
            print i, j, k
            if k == 3:
                goto .end # breaking out from a deeply nested loop
label .end
print "Finished"

Answer 68

Taking advantage of python's dynamic nature to have an apps config files in python syntax. For example if you had the following in a config file:

{
  "name1": "value1",
  "name2": "value2"
}

Then you could trivially read it like:

config = eval(open("filename").read())

Answer 69

Nested Function Parameter Re-binding

def create_printers(n):
    for i in xrange(n):
        def printer(i=i): # Doesn't work without the i=i
            print i
        yield printer

Answer 70

A slight misfeature of python. The normal fast way to join a list of strings together is,

''.join(list_of_strings)

Answer 71

import antigravity

Answer 72

Private methods and data hiding (encapsulation)

There's a common idiom in Python of denoting methods and other class members that are not intended to be part of the class's external API by giving them names that start with underscores. This is convenient and works very well in practice, but it gives the false impression that Python does not support true encapsulation of private code and/or data. In fact, Python automatically gives you lexical closures, which make it very easy to encapsulate data in a much more bulletproof way when the situation really warrants it. Here's a contrived example of a class that makes use of this technique:

class MyClass(object):
  def __init__(self):

    privateData = {}

    self.publicData = 123

    def privateMethod(k):
      print privateData[k] + self.publicData

    def privilegedMethod():
      privateData['foo'] = "hello "
      privateMethod('foo')

    self.privilegedMethod = privilegedMethod

  def publicMethod(self):
    print self.publicData

And here's a contrived example of its use:

>>> obj = MyClass()
>>> obj.publicMethod()
123
>>> obj.publicData = 'World'
>>> obj.publicMethod()
World
>>> obj.privilegedMethod()
hello World
>>> obj.privateMethod()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'MyClass' object has no attribute 'privateMethod'
>>> obj.privateData
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'MyClass' object has no attribute 'privateData'

The key is that privateMethod and privateData aren't really attributes of obj at all, so they can't be accessed from outside, nor do they show up in dir() or similar. They're local variables in the constructor, completely inaccessible outside of __init__. However, because of the magic of closures, they really are per-instance variables with the same lifetime as the object with which they're associated, even though there's no way to access them from outside except (in this example) by invoking privilegedMethod. Often this sort of very strict encapsulation is overkill, but sometimes it really can be very handy for keeping an API or a namespace squeaky clean.

In Python 2.x, the only way to have mutable private state is with a mutable object (such as the dict in this example). Many people have remarked on how annoying this can be. Python 3.x will remove this restriction by introducing the nonlocal keyword described in PEP 3104.

Answer 73

Using keyword arguments as assignments

Sometimes one wants to build a range of functions depending on one or more parameters. However this might easily lead to closures all referring to the same object and value:

funcs = [] 
for k in range(10):
     funcs.append( lambda: k)

>>> funcs[0]()
9
>>> funcs[7]()
9

This behaviour can be avoided by turning the lambda expression into a function depending only on its arguments. A keyword parameter stores the current value that is bound to it. The function call doesn't have to be altered:

funcs = [] 
for k in range(10):
     funcs.append( lambda k = k: k)

>>> funcs[0]()
0
>>> funcs[7]()
7

Answer 74

Method replacement for object instance

You can replace methods of already created object instances. It allows you to create object instance with different (exceptional) functionality:

>>> class C(object):
...     def fun(self):
...         print "C.a", self
...
>>> inst = C()
>>> inst.fun()  # C.a method is executed
C.a <__main__.C object at 0x00AE74D0>
>>> instancemethod = type(C.fun)
>>>
>>> def fun2(self):
...     print "fun2", self
...
>>> inst.fun = instancemethod(fun2, inst, C)  # Now we are replace C.a by fun2
>>> inst.fun()  # ... and fun2 is executed
fun2 <__main__.C object at 0x00AE74D0>

As we can C.a was replaced by fun2() in inst instance (self didn't change).

Alternatively we may use new module, but it's depreciated since Python 2.6:

>>> def fun3(self):
...     print "fun3", self
...
>>> import new
>>> inst.fun = new.instancemethod(fun3, inst, C)
>>> inst.fun()
fun3 <__main__.C object at 0x00AE74D0>

Node: This solution shouldn't be used as general replacement of inheritance mechanism! But it may be very handy in some specific situations (debugging, mocking).

Warning: This solution will not work for built-in types and for new style classes using slots.

Answer 75

Referencing a list comprehension as it is being built...

You can reference a list comprehension as it is being built by the symbol '_[1]'. For example, the following function unique-ifies a list of elements without changing their order by referencing its list comprehension.

def unique(my_list):
    return [x for x in my_list if x not in locals()['_[1]']]

Answer 76

set/frozenset

Probably an easily overlooked python builtin is "set/frozenset".

Useful when you have a list like this, [1,2,1,1,2,3,4] and only want the uniques like this [1,2,3,4].

Using set() that's exactly what you get:

>>> x = [1,2,1,1,2,3,4] 
>>> 
>>> set(x) 
set([1, 2, 3, 4]) 
>>>
>>> for i in set(x):
...     print i
...
1
2
3
4

And of course to get the number of uniques in a list:

>>> len(set([1,2,1,1,2,3,4]))
4

You can also find if a list is a subset of another list using, suprise, set().isasubset()

>>> set([1,2,3,4]).isasubset([0,1,2,3,4,5])
True

For more details: http://docs.python.org/library/stdtypes.html#set

Answer 77

Functional support.

Generators and generator expressions, specifically.

Ruby made this mainstream again, but Python can do it just as well. Not as ubiquitous in the libraries as in Ruby, which is too bad, but I like the syntax better, it's simpler.

Because they're not as ubiquitous, I don't see as many examples out there on why they're useful, but they've allowed me to write cleaner, more efficient code.

Answer 78

While debugging complex data structures pprint module comes handy.

Quoting from the docs..

>>> import pprint    
>>> stuff = sys.path[:]
>>> stuff.insert(0, stuff)
>>> pprint.pprint(stuff)
[<Recursion on list with id=869440>,
 '',
 '/usr/local/lib/python1.5',
 '/usr/local/lib/python1.5/test',
 '/usr/local/lib/python1.5/sunos5',
 '/usr/local/lib/python1.5/sharedmodules',
 '/usr/local/lib/python1.5/tkinter']

Answer 79

>>> from functools import partial
>>> bound_func = partial(range, 0, 10)
>>> bound_func()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> bound_func(2)
[0, 2, 4, 6, 8]

not really a hidden feature but partial is extremely useful for having late evaluation of functions.

you can bind as many or as few parameters in the initial call to partial as you want, and call it with any remaining parameters later (in this example i've bound the begin/end args to range, but call it the second time with a step arg)

Answer 80

is_ok() and "Yes" or "No"

Answer 81

...that dict.get() has a default value of None, thereby avoiding KeyErrors:

In [1]: test = { 1 : 'a' }

In [2]: test[2]
---------------------------------------------------------------------------
              Traceback (most recent call last)

<ipython console> in ()

: 2

In [3]: test.get( 2 )

In [4]: test.get( 1 )
Out[4]: 'a'

In [5]: test.get( 2 ) == None
Out[5]: True

and even to specify this 'at the scene':

In [6]: test.get( 2, 'Some' ) == 'Some'
Out[6]: True

Answer 82

You can easily transpose an array with zip.

a = [(1,2), (3,4), (5,6)]
zip(*a)
# [(1, 3, 5), (2, 4, 6)]

Answer 83

Negative round

The round() function rounds a float number to given precision in decimal digits, but precision can be negative:

>>> str(round(1234.5678, -2))
'1200.0'
>>> str(round(1234.5678, 2))
'1234.57'

Note: round() always returns a float, str() used in the above example because floating point math is inexact, and under 2.x the second example can print as 1234.5700000000001. Also see the decimal module.

Answer 84

An interpreter within the interpreter

The standard library's code module let's you include your own read-eval-print loop inside a program, or run a whole nested interpreter. E.g. (copied my example from here)

$ python
Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> shared_var = "Set in main console"
>>> import code
>>> ic = code.InteractiveConsole({ 'shared_var': shared_var })
>>> try:
...     ic.interact("My custom console banner!")
... except SystemExit, e:
...     print "Got SystemExit!"
... 
My custom console banner!
>>> shared_var
'Set in main console'
>>> shared_var = "Set in sub-console"
>>> sys.exit()
Got SystemExit!
>>> shared_var
'Set in main console'

This is extremely useful for situations where you want to accept scripted input from the user, or query the state of the VM in real-time.

TurboGears uses this to great effect by having a WebConsole from which you can query the state of you live web app.

Answer 85

Operator overloading for the set builtin:

>>> a = set([1,2,3,4])
>>> b = set([3,4,5,6])
>>> a | b # Union
{1, 2, 3, 4, 5, 6}
>>> a & b # Intersection
{3, 4}
>>> a < b # Subset
False
>>> a - b # Difference
{1, 2}
>>> a ^ b # Symmetric Difference
{1, 2, 5, 6}

More detail from the standard library reference: Set Types

Answer 86

You can override the mro of a class with a metaclass

>>> class A(object):
...     def a_method(self):
...         print("A")
... 
>>> class B(object):
...     def b_method(self):
...         print("B")
... 
>>> class MROMagicMeta(type):
...     def mro(cls):
...         return (cls, B, object)
... 
>>> class C(A, metaclass=MROMagicMeta):
...     def c_method(self):
...         print("C")
... 
>>> cls = C()
>>> cls.c_method()
C
>>> cls.a_method()
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
AttributeError: 'C' object has no attribute 'a_method'
>>> cls.b_method()
B
>>> type(cls).__bases__
(<class '__main__.A'>,)
>>> type(cls).__mro__
(<class '__main__.C'>, <class '__main__.B'>, <class 'object'>)

It's probably hidden for a good reason. :)

Answer 87

The reversed() builtin. It makes iterating much cleaner in many cases.

quick example:

for i in reversed([1, 2, 3]):
    print(i)

produces:

3
2
1

However, reversed() also works with arbitrary iterators, such as lines in a file, or generator expressions.

Answer 88

The Zen of Python

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Answer 89

pdb — The Python Debugger

As a programmer, one of the first things that you need for serious program development is a debugger. Python has one built-in which is available as a module called pdb (for "Python DeBugger", naturally!).

http://docs.python.org/library/pdb.html

Answer 90

Objects of small intgers (-5 .. 256) never created twice:

>>> a1 = -5; b1 = 256
>>> a2 = -5; b2 = 256
>>> id(a1) == id(a2), id(b1) == id(b2)
(True, True)
>>>
>>> c1 = -6; d1 = 257
>>> c2 = -6; d2 = 257
>>> id(c1) == id(c2), id(d1) == id(d2)
(False, False)
>>>

Edit: List objects never destroyed (only objects in lists). Python has array in which it keeps up to 80 empty lists. When you destroy list object - python puts it to that array and when you create new list - python gets last puted list from this array:

>>> a = [1,2,3]; a_id = id(a)
>>> b = [1,2,3]; b_id = id(b)
>>> del a; del b
>>> c = [1,2,3]; id(c) == b_id
True
>>> d = [1,2,3]; id(d) == a_id
True
>>>

Answer 91

Creating dictionary of two sequences that have related data

In [15]: t1 = (1, 2, 3)

In [16]: t2 = (4, 5, 6)

In [17]: dict (zip(t1,t2))
Out[17]: {1: 4, 2: 5, 3: 6}

Answer 92

Simulating the tertiary operator using and and or.

and and or operators in python return the objects themselves rather than Booleans. Thus:

In [18]: a = True

In [19]: a and 3 or 4
Out[19]: 3

In [20]: a = False

In [21]: a and 3 or 4
Out[21]: 4

However, Py 2.5 seems to have added an explicit tertiary operator

    In [22]: a = 5 if True else '6'

    In [23]: a
    Out[23]: 5

Well, this works if you are sure that your true clause does not evaluate to False. example:

>>> def foo(): 
...     print "foo"
...     return 0
...
>>> def bar(): 
...     print "bar"
...     return 1
...
>>> 1 and foo() or bar()
foo
bar
1

To get it right, you've got to just a little bit more:

>>> (1 and [foo()] or [bar()])[0]
foo
0

However, this isn't as pretty. if your version of python supports it, use the conditional operator.

>>> foo() if True or bar()
foo
0

Answer 93

inspect module is also a cool feature.

Answer 94

The spam module in standard Python

It is used for testing purposes.

I've picked it from ctypes tutorial. Try it yourself:

>>> import __hello__
Hello world...
>>> type(__hello__)
<type 'module'>
>>> from __phello__ import spam
Hello world...
Hello world...
>>> type(spam)
<type 'module'>
>>> help(spam)
Help on module __phello__.spam in __phello__:

NAME
    __phello__.spam

FILE
    c:\python26\<frozen>

Answer 95

Memory Management

Python dynamically allocates memory and uses garbage collection to recover unused space. Once an object is out of scope, and no other variables reference it, it will be recovered. I do not have to worry about buffer overruns and slowly growing server processes. Memory management is also a feature of other dynamic languages but Python just does it so well.

Of course, we must watch out for circular references and keeping references to objects which are no longer needed, but weak references help a lot here.

Answer 96

re can call functions!

The fact that you can call a function every time something matches a regular expression is very handy. Here I have a sample of replacing every "Hello" with "Hi," and "there" with "Fred", etc.

import re

def Main(haystack):
  # List of from replacements, can be a regex
  finds = ('Hello', 'there', 'Bob')
  replaces = ('Hi,', 'Fred,', 'how are you?')

  def ReplaceFunction(matchobj):
    for found, rep in zip(matchobj.groups(), replaces):
      if found != None:
        return rep

    # log error
    return matchobj.group(0)

  named_groups = [ '(%s)' % find for find in finds ]
  ret = re.sub('|'.join(named_groups), ReplaceFunction, haystack)
  print ret

if __name__ == '__main__':
  str = 'Hello there Bob'
  Main(str)
  # Prints 'Hi, Fred, how are you?'

Answer 97

I personally love the 3 different quotes

str = "I'm a string 'but still I can use quotes' inside myself!"
str = """ For some messy multi line strings.
Such as
<html>
<head> ... </head>"""

Also cool: not having to escape regular expressions, avoiding horrible backslash salad by using raw strings:

str2 = r"\n" 
print str2
>> \n

And my favourite:

Getting values from a dict, without having to worry if the key exists, and it even sets the key for you! (I love you Python guys!)

The 3 times happiness dict package:

a = {}
print a.setdefault("mykey",20) 
# Prints value of a['mykey'] if key exists.
# Prints 20, if key doesn't exist.
# And even adds 20 to the dict in that case.
# This has made so many parts of my code so much nicer!

Answer 98

One word: IPython

Tab introspection, pretty-printing, %debug, history management, pylab, ... well worth the time to learn well.

Answer 99

Reloading modules enables a "live-coding" style. But class instances don't update. Here's why, and how to get around it. Remember, everything, yes, everything is an object.

>>> from a_package import a_module
>>> cls = a_module.SomeClass
>>> obj = cls()
>>> obj.method()
(old method output)

Now you change the method in a_module.py and want to update your object.

>>> reload(a_module)
>>> a_module.SomeClass is cls
False # Because it just got freshly created by reload.
>>> obj.method()
(old method output)

Here's one way to update it (but consider it running with scissors):

>>> obj.__class__ is cls
True # it's the old class object
>>> obj.__class__ = a_module.SomeClass # pick up the new class
>>> obj.method()
(new method output)

This is "running with scissors" because the object's internal state may be different than what the new class expects. This works for really simple cases, but beyond that, pickle is your friend. It's still helpful to understand why this works, though.

Answer 100

Not very hidden, but functions have attributes:

def doNothing():
    pass

doNothing.monkeys = 4
print doNothing.monkeys
4

Answer 101

You can decorate functions with classes - replacing the function with a class instance:

class countCalls(object):
    """ decorator replaces a function with a "countCalls" instance
    which behaves like the original function, but keeps track of calls

    >>> @countCalls
    ... def doNothing():
    ...     pass
    >>> doNothing()
    >>> doNothing()
    >>> print doNothing.timesCalled
    2
    """
    def __init__ (self, functionToTrack):
        self.functionToTrack = functionToTrack
        self.timesCalled = 0
    def __call__ (self, *args, **kwargs):
        self.timesCalled += 1
        return self.functionToTrack(*args, **kwargs)

Answer 102

With a minute amount of work, the threading module becomes amazingly easy to use. This decorator changes a function so that it runs in its own thread, returning a placeholder class instance instead of its regular result. You can probe for the answer by checking placeolder.result or wait for it by calling placeholder.awaitResult()

def threadify(function):
    """
    exceptionally simple threading decorator. Just:
    >>> @threadify
    ... def longOperation(result):
    ...     time.sleep(3)
    ...     return result
    >>> A= longOperation("A has finished")
    >>> B= longOperation("B has finished")

    A doesn't have a result yet:
    >>> print A.result
    None

    until we wait for it:
    >>> print A.awaitResult()
    A has finished

    we could also wait manually - half a second more should be enough for B:
    >>> time.sleep(0.5); print B.result
    B has finished
    """
    class thr (threading.Thread,object):
        def __init__(self, *args, **kwargs):
            threading.Thread.__init__ ( self )  
            self.args, self.kwargs = args, kwargs
            self.result = None
            self.start()
        def awaitResult(self):
            self.join()
            return self.result        
        def run(self):
            self.result=function(*self.args, **self.kwargs)
    return thr

Answer 103

ROT13 is a valid encoding for source code, when you use the right coding declaration at the top of the code file:

#!/usr/bin/env python
# -*- coding: rot13 -*-

cevag "Uryyb fgnpxbiresybj!".rapbqr("rot13")

Answer 104

If you've renamed a class in your application where you're loading user-saved files via Pickle, and one of the renamed classes are stored in a user's old save, you will not be able to load in that pickled file.

However, simply add in a reference to your class definition and everything's good:

e.g., before:

class Bleh:
    pass

now,

class Blah:
    pass

so, your user's pickled saved file contains a reference to Bleh, which doesn't exist due to the rename. The fix?

Bleh = Blah

simple!

Answer 105

The fact that EVERYTHING is an object, and as such is extensible. I can add member variables as metadata to a function that I define:

>>> def addInts(x,y): 
...    return x + y
>>> addInts.params = ['integer','integer']
>>> addInts.returnType = 'integer'

This can be very useful for writing dynamic unit tests, e.g.

Answer 106

The getattr built-in function :

>>> class C():
    def getMontys(self):
     self.montys = ['Cleese','Palin','Idle','Gilliam','Jones','Chapman']
     return self.montys


>>> c = C()
>>> getattr(c,'getMontys')()
['Cleese', 'Palin', 'Idle', 'Gilliam', 'Jones', 'Chapman']
>>>

Useful if you want to dispatch function depending on the context. See examples in Dive Into Python (Here)

Answer 107

Simple way to test if a key is in a dict:

>>> 'key' in { 'key' : 1 }
True

>>> d = dict(key=1, key2=2)
>>> if 'key' in d:
...     print 'Yup'
... 
Yup

Answer 108

Classes as first-class objects (shown through a dynamic class definition)

Note the use of the closure as well. If this particular example looks like a "right" approach to a problem, carefully reconsider ... several times :)

def makeMeANewClass(parent, value):
  class IAmAnObjectToo(parent):
    def theValue(self):
      return value
  return IAmAnObjectToo

Klass = makeMeANewClass(str, "fred")
o = Klass()
print isinstance(o, str)  # => True
print o.theValue()        # => fred

Answer 109

Exposing Mutable Buffers

Using the Python Buffer Protocol to expose mutable byte-oriented buffers in Python (2.5/2.6).

(Sorry, no code here. Requires use of low-level C API or existing adapter module).

Answer 110

Extending properties (defined as descriptor) in subclasses

Sometimes it's useful to extent (modify) value "returned" by descriptor in subclass. It can be easily done with super():

class A(object):
    @property
    def prop(self):
        return {'a': 1}

class B(A):
    @property
    def prop(self):
        return dict(super(B, self).prop, b=2)

Store this in test.py and run python -i test.py (another hidden feature: -i option executed the script and allow you to continue in interactive mode):

>>> B().prop
{'a': 1, 'b': 2}

Answer 111

The pythonic idiom x = ... if ... else ... is far superior to x = ... and ... or ... and here is why:

Although the statement

x = 3 if (y == 1) else 2

Is equivalent to

x = y == 1 and 3 or 2

if you use the x = ... and ... or ... idiom, some day you may get bitten by this tricky situation:

x = 0 if True else 1    # sets x equal to 0

and therefore is not equivalent to

x = True and 0 or 1   # sets x equal to 1

For more on the proper way to do this, see http://stackoverflow.com/questions/101268/hidden-features-of-python/116480#116480.

Answer 112

Python can understand any kind of unicode digits, not just the ASCII kind:

>>> s = u'１０５８５'
>>> s
u'\uff11\uff10\uff15\uff18\uff15'
>>> print s
１０５８５
>>> int(s)
10585
>>> float(s)
10585.0

Answer 113

Regarding Nick Johnson's implementation of a Property class (just a demonstration of descriptors, of course, not a replacement for the built-in), I'd include a setter that raises an AttributeError:

class Property(object):
    def __init__(self, fget):
        self.fget = fget

    def __get__(self, obj, type):
        if obj is None:
            return self
        return self.fget(obj)

    def __set__(self, obj, value):
       raise AttributeError, 'Read-only attribute'

Including the setter makes this a data descriptor as opposed to a method/non-data descriptor. A data descriptor has precedence over instance dictionaries. Now an instance can't have a different object assigned to the property name, and attempts to assign to the property will raise an attribute error.

Answer 114

The unpacking syntax has been upgraded in the recent version as can be seen in the example.

>>> a, *b = range(5)
>>> a, b
(0, [1, 2, 3, 4])
>>> *a, b = range(5)
>>> a, b
([0, 1, 2, 3], 4)
>>> a, *b, c = range(5)
>>> a, b, c
(0, [1, 2, 3], 4)

Answer 115

Manipulating Recursion Limit

Getting or setting the maximum depth of recursion with sys.getrecursionlimit() & sys.setrecursionlimit().

We can limit it to prevent a stack overflow caused by infinite recursion.

Answer 116

Multiplying by a boolean

One thing I'm constantly doing in web development is optionally printing HTML parameters. We've all seen code like this in other languages:

class='<% isSelected ? "selected" : "" %>'

In Python, you can multiply by a boolean and it does exactly what you'd expect:

class='<% "selected" * isSelected %>'

This is because multiplication coerces the boolean to an integer (0 for False, 1 for True), and in python multiplying a string by an int repeats the string N times.

Answer 117

Mod works correctly with negative numbers

-1 % 5 is 4, as it should be, not -1 as it is in other languages like JavaScript. This makes "wraparound windows" cleaner in Python, you just do this:

index = (index + increment) % WINDOW_SIZE

Answer 118

You can construct a functions kwargs on demand:

kwargs = {}
kwargs[str("%s__icontains" % field)] = some_value
some_function(**kwargs)

The str() call is somehow needed, since python complains otherwise that it is no string. Don't know why ;) I use this for a dynamic filters within Djangos object model:

result = model_class.objects.filter(**kwargs)

Answer 119

Guessing integer base

>>> int('10', 0)
10
>>> int('0x10', 0)
16
>>> int('010', 0)  # does not work on Python 3.x
8
>>> int('0o10', 0)  # Python >=2.6 and Python 3.x
8
>>> int('0b10', 0)  # Python >=2.6 and Python 3.x
2

Answer 120

itertools

This module is often overlooked. The following example uses itertools.chain() to flatten a list:

>>> from itertools import *
>>> l = [[1, 2], [3, 4]]
>>> list(chain(*l))
[1, 2, 3, 4]

See http://docs.python.org/library/itertools.html#recipes for more applications.

Answer 121

Monkeypatching objects

Every class in Python has a __dict__ object which stores the class's attributes. So, you can do something like this:

class Foo(object):
    def __init__(self, arg1, arg2, **kwargs):
        #do stuff with arg1 and arg2
        self.__dict__.update(kwargs)

f = Foo('arg1', 'arg2', bar=20, baz=10)
#now f is a Foo object with two extra attributes

This can be exploited to add both attributes and functions arbitrarily to objects. This can also be exploited to create a quick-and-dirty struct type.

class struct(object):
    def __init__(**kwargs):
       self.__dict__.update(kwargs)

s = struct(foo=10, bar=11, baz="i'm a string!')

Answer 122

Creating enums

In Python, you can do this to quickly create an enumeration:

>>> FOO, BAR, BAZ = range(3)
>>> FOO
0

But the "enums" don't have to have integer values. You can even do this:

class Colors(object):
    RED, GREEN, BLUE, YELLOW = (255,0,0), (0,255,0), (0,0,255), (0,255,255)

#now Colors.RED is a 3-tuple that returns the 24-bit 8bpp RGB 
#value for saturated red

Answer 123

Manipulating sys.modules

You can manipulate the modules cache directly, making modules available or unavailable as you wish:

>>> import sys
>>> import ham
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named ham

# Make the 'ham' module available -- as a non-module object even!
>>> sys.modules['ham'] = 'ham, eggs, saussages and spam.'
>>> import ham
>>> ham
'ham, eggs, saussages and spam.'

# Now remove it again.
>>> sys.modules['ham'] = None
>>> import ham
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named ham

This works even for modules that are available, and to some extent for modules that already are imported:

>>> import os
# Stop future imports of 'os'.
>>> sys.modules['os'] = None
>>> import os
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named os
# Our old imported module is still available.
>>> os
<module 'os' from '/usr/lib/python2.5/os.pyc'>

As the last line shows, changing sys.modules only affects future import statements, not past ones, so if you want to affect other modules it's important to make these changes before you give them a chance to try and import the modules -- so before you import them, typically. None is a special value in sys.modules, used for negative caching (indicating the module was not found the first time, so there's no point in looking again.) Any other value will be the result of the import operation -- even when it is not a module object. You can use this to replace modules with objects that behave exactly like you want. Deleting the entry from sys.modules entirely causes the next import to do a normal search for the module, even if it was already imported before.

Answer 124

There are no secrets in Python ;)

Answer 125

Passing tuple to builtin functions

Much Python functions accept tuples, also it doesn't seem like. For example you want to test if your variable is a number, you could do:

if isinstance (number, float) or isinstance (number, int):  
   print "yaay"

But if you pass us tuple this looks much cleaner:

if isinstance (number, (float, int)):  
   print "yaay"

Answer 126

You can assign several variables to the same value

>>> foo = bar = baz = 1
>>> foo, bar, baz
(1, 1, 1)

Useful to initialize several variable to None, in a compact way.

Answer 127

threading.enumerate() gives access to all Thread objects in the system and sys._current_frames() returns the current stack frames of all threads in the system, so combine these two and you get Java style stack dumps:

def dumpstacks(signal, frame):
    id2name = dict([(th.ident, th.name) for th in threading.enumerate()])
    code = []
    for threadId, stack in sys._current_frames().items():
        code.append("\n# Thread: %s(%d)" % (id2name[threadId], threadId))
        for filename, lineno, name, line in traceback.extract_stack(stack):
            code.append('File: "%s", line %d, in %s' % (filename, lineno, name))
            if line:
                code.append("  %s" % (line.strip()))
    print "\n".join(code)

import signal
signal.signal(signal.SIGQUIT, dumpstacks)

Do this at the beginning of a multi-threaded python program and you get access to current state of threads at any time by sending a SIGQUIT. You may also choose signal.SIGUSR1 or signal.SIGUSR2.

See

Answer 128

Braces

def g():
    print 'hi!'

def f(): (
    g()
)

>>> f()
hi!

Answer 129

Top Secret Attributes

>>> class A(object): pass
>>> a = A()
>>> setattr(a, "can't touch this", 123)
>>> dir(a)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', "can't touch this"]
>>> a.can't touch this # duh
  File "<stdin>", line 1
    a.can't touch this
                     ^
SyntaxError: EOL while scanning string literal
>>> getattr(a, "can't touch this")
123
>>> setattr(a, "__class__.__name__", ":O")
>>> a.__class__.__name__
'A'
>>> getattr(a, "__class__.__name__")
':O'

Answer 130

Nice treatment of infinite recursion in dictionaries:

>>> a = {}
>>> b = {}
>>> a['b'] = b
>>> b['a'] = a
>>> print a
{'b': {'a': {...}}}

Answer 131

Multiple references to an iterator

You can create multiple references to the same iterator using list multiplication:

>>> i = (1,2,3,4,5,6,7,8,9,10) # or any iterable object
>>> iterators = [iter(i)] * 2
>>> iterators[0].next()
1
>>> iterators[1].next()
2
>>> iterators[0].next()
3

This can be used to group an iterable into chunks, for example, as in this example from the itertools documentation

def grouper(n, iterable, fillvalue=None):
    "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

Answer 132

You can ask any object which module it came from by looking at its __ module__ property. This is useful, for example, if you're experimenting at the command line and have imported a lot of things.

Along the same lines, you can ask a module where it came from by looking at its __ file__ property. This is useful when debugging path issues.

Answer 133

reversing an iterable using negative step

>>> s = "Hello World"
>>> s[::-1]
'dlroW olleH'
>>> a = (1,2,3,4,5,6)
>>> a[::-1]
(6, 5, 4, 3, 2, 1)
>>> a = [5,4,3,2,1]
>>> a[::-1]
[1, 2, 3, 4, 5]

Answer 134

When using the interactive shell, "_" contains the value of the last printed item:

>>> range(10)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> _
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

Answer 135

Slices & Mutability

Copying lists

>>> x = [1,2,3]
>>> y = x[:]
>>> y.pop()
3
>>> y
[1, 2]
>>> x
[1, 2, 3]

Replacing lists

>>> x = [1,2,3]
>>> y = x
>>> y[:] = [4,5,6]
>>> x
[4, 5, 6]

Answer 136

Combine unpacking with the print function:

# in 2.6 <= python < 3.0, 3.0 + the print function is native
from __future__ import print_function 

mylist = ['foo', 'bar', 'some other value', 1,2,3,4]  
print(*mylist)

Answer 137

Feature Listing

• Chaining Comparison Operators
• Regex Debugging
• Generator expressions
• Function as iter() argument
• Enumeration
• Sending to Generators
• Decorators
• missing items
• List stepping
• Multi-line Regex
• Braces ;)
• For/else
• In Place Value Swapping
• Default Argument Gotchas
• Docstring Tests
• with statement
• Descriptors
• Argument Unpacking
• Dictionary default .get value
• New types at runtime
• Using the Ternary Expression
• Named string formatting
• .pth files
• Tab Completion in Interactive Interpreter
• ROT13 Encoding
• import this
• try/except/else
• Nested list/generator comprehensions
• Ellipsis Slicing Syntax
• Unpacking+print() function

(note: This listing is not yet complete)

Answer 138

From python 3.1 ( 2.7 ) dictionary and set comprehensions are supported :

{ a:a for a in range(10) }
{ a for a in range(10) }

Answer 139

Python 2.x ignore commas if found after the last element of the "set"

>>> a_tuple_for_instance = (0,1,2,3)
>>> another_tuple = (0,1,2,3)
>>> a_tuple_for_instance == anther_tuple
True

be aware of the fact that a tuple with only one element need a comma

a_tuple_with_one_element = (8,)

Answer 140

** Using sets to reference contents in sets of frozensets**

As you probably know, sets are mutable and thus not hashable, so it's necessary to use frozensets if you want to make a set of sets (or use sets as dictionary keys):

>>> fabc = frozenset('abc')
>>> fxyz = frozenset('xyz')
>>> mset = set((fabc, fxyz))
>>> mset
{frozenset({'a', 'c', 'b'}), frozenset({'y', 'x', 'z'})}

However, it's possible to test for membership and remove/discard members using just ordinary sets:

>>> abc = set('abc')
>>> abc in mset
True
>>> mset.remove(abc)
>>> mset
{frozenset({'y', 'x', 'z'})}

To quote from the Python Standard Library docs:

Note, the elem argument to the __contains__(), remove(), and discard() methods may be a set. To support searching for an equivalent frozenset, the elem set is temporarily mutated during the search and then restored. During the search, the elem set should not be read or mutated since it does not have a meaningful value.

Unfortunately, and perhaps astonishingly, the same is not true of dictionaries:

>>> mdict = {fabc:1, fxyz:2}
>>> fabc in mdict
True
>>> abc in mdict
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
TypeError: unhashable type: 'set'

Answer 141

The textwrap.dedent utility function in python can come quite in handy testing that a multiline string returned is equal to the expected output without breaking the indentation of your unittests:

import unittest, textwrap

class XMLTests(unittest.TestCase):
    def test_returned_xml_value(self):
        returned_xml = call_to_function_that_returns_xml()
        expected_value = textwrap.dedent("""\
        <?xml version="1.0" encoding="utf-8"?>
        <root_node>
            <my_node>my_content</my_node>
        </root_node>
        """)

        self.assertEqual(expected_value, returned_xml)

Answer 142

Slices as lvalues. This Sieve of Eratosthenes produces a list that has either the prime number or 0. Elements are 0'd out with the slice assignment in the loop.

def eras(n):
    last = n + 1
    sieve = [0,0] + list(range(2, last))
    sqn = int(round(n ** 0.5))
    it = (i for i in xrange(2, sqn + 1) if sieve[i])
    for i in it:
        sieve[i*i:last:i] = [0] * (n//i - i + 1)
    return filter(None, sieve)

To work, the slice on the left must be assigned a list on the right of the same length.

Answer 143

Zero-argument and variable-argument lambdas

Lambda functions are usually used for a quick transformation of one value into another, but they can also be used to wrap a value in a function:

>>> f = lambda: 'foo'
>>> f()
'foo'

They can also accept the usual *args and **kwargs syntax:

>>> g = lambda *args, **kwargs: args[0], kwargs['thing']
>>> g(1, 2, 3, thing='stuff')
(1, 'stuff')

Answer 144

Multi line strings

One approach is to use backslashes:

>>> sql = "select * from some_table \
where id > 10"
>>> print sql
select * from some_table where id > 10

Another is to use the triple-quote:

>>> sql = """select * from some_table 
where id > 10"""
>>> print sql
select * from some_table where id > 10

Problem with those is that they are not indented (look poor in your code). If you try to indent, it'll just print the white-spaces you put.

A third solution, which I found about recently, is to divide your string into lines and surround with parentheses:

>>> sql = ("select * from some_table " # <-- no comma, whitespace at end
           "where id > 10 "
           "order by name") 
>>> print sql
select * from some_table where id > 10 order by name

note how there's no comma between lines (this is not a tuple), and you have to account for any trailing/leading white spaces that your string needs to have. All of these work with placeholders, by the way (such as "my name is %s" % name).

Answer 145

pow() can also calculate (x ** y) % z efficiently.

There is a lesser known third argument of the built-in pow() function that allows you to calculate x^y modulo z more efficiently than simply doing (x ** y) % z:

>>> x, y, z = 1234567890, 2345678901, 17
>>> pow(x, y, z)            # almost instantaneous
6

In comparison, (x ** y) % z didn't given a result in one minute on my machine for the same values.

Answer 146

Backslashes inside raw strings can still escape quotes. See this:

>>> print repr(r"aaa\"bbb")
'aaa\\"bbb'

Note that both the backslash and the double-quote are present in the final string.

As consequence, you can't end a raw string with a backslash:

>>> print repr(r"C:\")
SyntaxError: EOL while scanning string literal
>>> print repr(r"C:\"")
'C:\\"'

This happens because raw strings were implemented to help writing regular expressions, and not to write Windows paths. Read a long discussion about this at Gotcha — backslashes in Windows filenames.

Answer 147

Sequence multiplication and reflected operands

>>> 'xyz' * 3
'xyzxyzxyz'

>>> [1, 2] * 3
[1, 2, 1, 2, 1, 2]

>>> (1, 2) * 3
(1, 2, 1, 2, 1, 2)

We get the same result with reflected (swapped) operands

>>> 3 * 'xyz'
'xyzxyzxyz'

It works like this:

>>> s = 'xyz'
>>> num = 3

To evaluate an expression s * num interpreter calls s.__mul__(num)

>>> s * num
'xyzxyzxyz'

>>> s.__mul__(num)
'xyzxyzxyz'

To evaluate an expression num * s interpreter calls num.__mul__(s)

>>> num * s
'xyzxyzxyz'

>>> num.__mul__(s)
NotImplemented

If the call returns NotImplemented then interpreter calls a reflected operation s.__rmul__(num) if operands have different types

>>> s.__rmul__(num)
'xyzxyzxyz'

See http://docs.python.org/reference/datamodel.html#object.rmul

Answer 148

enumerate with different starting index

enumerate has partly been covered in this answer, but recently I've found an even more hidden feature of enumerate that I think deserves its own post instead of just a comment.

Since Python 2.6, you can specify a starting index to enumerate in its second argument:

>>> l = ["spam", "ham", "eggs"]
>>> list(enumerate(l))
>>> [(0, "spam"), (1, "ham"), (2, "eggs")]
>>> list(enumerate(l, 1))
>>> [(1, "spam"), (2, "ham"), (3, "eggs")]

One place where I've found it utterly useful is when I am enumerating over entries of a symmetric matrix. Since the matrix is symmetric, I can save time by iterating over the upper triangle only, but in that case, I have to use enumerate with a different starting index in the inner for loop to keep track of the row and column indices properly:

for ri, row in enumerate(matrix):
    for ci, column in enumerate(matrix[ri:], ri):
        # ci now refers to the proper column index

Strangely enough, this behaviour of enumerate is not documented in help(enumerate), only in the online documentation.