Python Exceptions: EAFP and What is Really Exceptional?

Question

It's been said in a couple places (here and here) that Python's emphasis on "it's easier to ask for forgiveness than permission" (EAFP) should be tempered with the idea that exceptions should only be called in truly exceptional cases. Consider the following, in which we're popping and pushing on a priority queue until only one element is left:

import heapq
...
pq = a_list[:]
heapq.heapify(pq)
while True:
    min1 = heapq.heappop(pq)
    try:
        min2 = heapq.heappop(pq)
    except IndexError:
        break
    else
        heapq.heappush(pq, min1 + min2)
# do something with min1

The exception is only raised once in len(a_list) iterations of the loop, but it's not really exceptional, because we know its going to happen eventually. This setup saves us from checking whether a_list is empty a bunch of times, but (maybe) it's less readable than using explicit conditions.

What's the consensus on using exceptions for this kind of non-exceptional program logic?

Answer 1

Looking at the docs I think you can safely re-write the function as follows:

import heapq
...
pq = heapq.heapify(a_list)
while pq:
    min1 = heapq.heappop(pq)
    if pq:
        min2 = heapq.heappop(pq)
        heapq.heappush(pq, min1 + min2)
# do something with min1

..and thereby avoid the try-except.

Getting to the end of a list which is something you know is going to happen here isn't exceptional - it's garaunteed! So better practice would be to handle it in advance. If you had something else in another thread which was consuming from the same heap then using try-except there would make a lot more sense (i.e. handling a special / unpredictable case).

More generally, I would avoid try-excepts wherever I can test for and avoid a failure in advance. This forces you to say "I know this bad situation might happen so here's how I deal with it". In my opinion, you'll tend to write more readable code as a result.

[Edit] Updated the example as per Alex's suggestion

Answer 2

I've found the practice of using exceptions as "normal" flow control tools to be a fairly widely accepted in Python. It's most commonly used in situations like the one you describe, when you get to the end of some sort of sequence.

In my opinion, that's a perfectly valid use of an exception. You do want to be careful about using exception handling willy-nilly though. Raising an exception is a reasonably expensive operation, and therefore it's best to ensure you only rely on an exception at the end of the sequence, not in each iteration.

Answer 3

exceptions should only be called in truly exceptional cases

Not in Python: for example, every for loop (unless it prematurely breaks or returns) terminates by an exception (StopIteration) being thrown and caught. So, an exception that happens once per loop is hardly strange to Python -- it's there more often than not!

The principle in question may be crucial in other languages, but that's definitely no reason to apply that principle to Python, where it's so contrary to the language's ethos.

In this case I like Jon's rewrite (which should be further simplified by removing the else branch) because it makes the code more compact -- a pragmatical reason, most definitely not the "tempering" of Python style with an alien principle.

Answer 4

Throwing exceptions is expensive in most low-level languages like C++. That influences a lot of the "common wisdom" about exceptions, and doesn't apply so much to languages that run in a VM, like Python. There's not such a major cost in Python for using an exception instead of a conditional.

(This is a case where the "common wisdom" becomes a matter of habit. People come to it from experience in one type of environment--low-level languages--and then apply it to new domains without evaluating whether it makes sense.)

Exceptions are still, in general, exceptional. That doesn't mean that they don't happen often; it means that they're the exception. They're the things that will tend to break from ordinary code flow, and which most of the time you don't want to have to handle one by one--which is the point of exception handlers. This part is the same in Python as in C++ and all other languages with exceptions.

However, that tends to define when exceptions are thrown. You're talking about when exceptions should be caught. Very simply, don't worry about it: exceptions aren't expensive, so don't go to lengths to try to prevent them from being thrown. A lot of Python code is designed around this.

I don't agree with Jon's suggestion to try to test for and avoid exceptions in advance. That's fine if it leads to clearer code, as in his example. However, in many cases it's just going to complicate things--it can effectively lead to duplicating checks and introducing bugs. For example,

import os, errno, stat

def read_file(fn):
    """
    Read a file and return its contents.  If the file doesn't exist or
    can't be read, return "".
    """
    try:
        return open(fn).read()
    except IOError, e:
        return ""

def read_file_2(fn):
    """
    Read a file and return its contents.  If the file doesn't exist or
    can't be read, return "".
    """
    if not os.access(fn, os.R_OK):
        return ""
    st = os.stat(fn)
    if stat.S_ISDIR(st.st_mode):
        return ""
    return open(fn).read()

print read_file("x")

Sure, we can test for and avoid the failure--but we've complicated things badly. We're trying to guess all the ways the file access might fail (and this doesn't catch all of them), we may have introduced race conditions, and we're doing a lot more I/O work. This is all done for us--just catch the exception.

Answer 5

Just for the record, i'd write is like this:

import heapq
a_list = range(20)
pq = a_list[:]
heapq.heapify(pq)
try:
    while True:
        min1 = heapq.heappop(pq)
        min2 = heapq.heappop(pq)
        heapq.heappush(pq, min1 + min2)
except IndexError:
    pass # we ran out of numbers in pq

Exceptions can leave a loop (even functions) and you can use them for that. Since Python throws them everywhere, I think this pattern is quite useful (even pythonic).