Cheap exception handling in Python?

Question

I read in an earlier answer that exception handling is cheap in Python so we shouldn't do pre-conditional checking.

I have not heard of this before, but I'm relatively new to Python. Exception handling means a dynamic call and a static return, whereas an if statement is static call, static return.

How can doing the checking be bad and the try-except be good, seems to be the other way around. Can someone explain this to me?

Answer 1

You might find this post helpful: Try / Except Performance in Python: A Simple Test where Patrick Altman did some simple testing to see what the performance is in various scenarios pre-conditional checking (specific to dictionary keys in this case) and using only exceptions. Code is provided as well if you want to adapt it to test other conditionals.

The conclusions he came to:

From these results, I think it is fair to quickly determine a number of conclusions:

1. If there is a high likelihood that the element doesn't exist, then you are better off checking for it with has_key.
2. If you are not going to do anything with the Exception if it is raised, then you are better off not putting one have the except
3. If it is likely that the element does exist, then there is a very slight advantage to using a try/except block instead of using has_key, however, the advantage is very slight.

Answer 2

I am a python beginner as well. While I cannot say why exactly Exception handling has been called cheap in the context of that answer, here are my thoughts:

Note that checking with if-elif-else has to evaluate a condition every time. Exception handling, including the search for an exception handler occurs only in an exceptional condition, which is likely to be rare in most cases. That is a clear efficiency gain. As pointed out by Jay, it is better to use conditional logic rather than exceptions when there is a high likelihood of the key being absent. This is because if the key is absent most of the time, it is not an exceptional condition.

That said, I suggest that you don't worry about efficiency and rather about meaning. Use exception handling to detect exceptional cases and checking conditions when you want to decide upon something. I was reminded about the importance of meaning by S.Lott just yesterday.

Case in point:

def xyz(key):
   dictOb = {x:1, y:2, z:3}
   #Condition evaluated every time
   if dictOb.has_key(key):  #Access 1 to dict
        print dictOb[key]  #Access 2

Versus

#Exception mechanism is in play only when the key isn't found.
def xyz(key):
   dictOb = {x:1, y:2, z:3}
   try:
        print dictOb[key]  #Access 1
   except KeyError:
        print "Not Found"

Overall, having some code that handles something,like a missing key, just in case needs exception handling, but in situations like when the key isn't present most of the time, what you really want to do is to decide if the key is present => if-else. Python emphasizes and encourages saying what you mean.

Why why Exceptions are preferred to if-elif ->

1. It expresses the meaning more clearly when you are looking foe exceptional aka unusual/unexpected conditions in your code.
2. It is cleaner and a whole lot more readable.
3. It is more flexible.
4. It can be used to write more concise code.
5. Avoids a lot of nasty checking.
6. It is more maintainable.

Note When we avoid using try-except, Exceptions continue being raised. Exceptions which aren't handled simply go to the default handler. When you use try-except, you can handle the error yourself. It might be more efficient because if-else requires condition evaluation, while looking for an exception handler may be cheaper. Even if this is true, the gain from it will be too minor to bother thinking about.

I hope my answer helps.

Answer 3

What are static versus dynamic calls and returns, and why do you think that calls and returns are any different in Python depending on if you are doing it in a try/except block? Even if you aren't catching an exception, Python still has to handle the call possibly raising something, so it doesn't make a difference to Python in regards to how the calls and returns are handled.

Every function call in Python involves pushing the arguments onto the stack, and invoking the callable. Every single function termination is followed by the caller, in the internal wiring of Python, checking for a successful or exception termination, and handles it accordingly. In other words, if you think that there is some additional handling when you are in a try/except block that is somehow skipped when you are not in one, you are mistaken. I assume that is what you "static" versus "dynamic" distinction was about.

Further, it is a matter of style, and experienced Python developers come to read exception catching well, so that when they see the appropriate try/except around a call, it is more readable than a conditional check.

Answer 4

With Python, it is easy to check different possibilities for speed - get to know the timeit module :

... example session (using the command line) that compare the cost of using hasattr() vs. try/except to test for missing and present object attributes.

% timeit.py 'try:' '  str.__nonzero__' 'except AttributeError:' '  pass'
100000 loops, best of 3: 15.7 usec per loop
% timeit.py 'if hasattr(str, "__nonzero__"): pass'
100000 loops, best of 3: 4.26 usec per loop
% timeit.py 'try:' '  int.__nonzero__' 'except AttributeError:' '  pass'
1000000 loops, best of 3: 1.43 usec per loop
% timeit.py 'if hasattr(int, "__nonzero__"): pass'
100000 loops, best of 3: 2.23 usec per loop

EDIT: The command line option -n will default to a large enough count so that the run time is meaningful. A quote from the manual:

If -n is not given, a suitable number of loops is calculated by trying successive powers of 10 until the total time is at least 0.2 seconds.

Answer 5

Don't sweat the small stuff. You've already picked one of the slower scripting languages out there, so trying to optimize down to the opcode is not going to help you much. The reason to choose an interpreted, dynamic language like Python is to optimize your time, not the CPU's.

If you use common language idioms, then you'll see all the benefits of fast prototyping and clean design and your code will naturally run faster as new versions of Python are released and the computer hardware is upgraded.

If you have performance problems, then profile your code and optimize your slow algorithms. But in the mean time, use exceptions for exceptional situations since it will make any refactoring you ultimately do along these lines a lot easier.

Answer 6

"Can someone explain this to me?"

Depends.

Here's one explanation, but it's not helpful. Your question stems from your assumptions. Since the real world conflicts with your assumptions, it must mean your assumptions are wrong. Not much of an explanation, but that's why you're asking.

"Exception handling means a dynamic call and a static return, whereas an if statement is static call, static return."

What does "dynamic call" mean? Searching stack frames for a handler? I'm assuming that's what you're talking about. And a "static call" is somehow locating the block after the if statement.

Perhaps this "dynamic call" is not the most costly part of the operation. Perhaps the if-statement expression evaluation is slightly more expensive than the simpler "try-it-and-fail".

Turns out that Python's internal integrity checks are almost the same as your if-statement, and have to be done anyway. Since Python's always going to check, your if-statement is (mostly) redundant.

You can read about low-level exception handling in http://docs.python.org/c-api/intro.html#exceptions.

---

Edit

More to the point: The if vs. except debate doesn't matter.

Since exceptions are cheap, do not label them as a performance problem.

Use what makes your code clear and meaningful. Don't waste time on micro-optimizations like this.

Answer 7

Putting aside the performance measurements that others have said, the guiding principle is often structured as "it is easier to ask forgiveness than ask permission" vs. "look before you leap."

Consider these two snippets:

# Look before you leap
if not os.path.exists(filename):
    raise SomeError("Cannot open configuration file")
f = open(filename)

vs.

# Ask forgiveness ...
try:
  f = open(filename)
except IOError:
  raise SomeError("Cannot open configuration file")

Equivalent? Not really. OSes are multi-taking systems. What happens if the file was deleted between the test for 'exists' and 'open' call?

What happens if the file exists but it's not readable? What if it's a directory name instead of a file. There can be many possible failure modes and checking all of them is a lot of work. Especially since the 'open' call already checks and reports all of those possible failures.

The guideline should be to reduce the chance of inconsistent state, and the best way for that is to use exceptions instead of test/call.

Answer 8

The general message, as S.Lott said, is that try/except doesn't hurt so you should feel free to use it whenever it seems appropriate.

This debate is often called "LBYL vs EAFP" – that's "look before you leap" vs "easier to ask forgiveness than permission". Alex Martelli weighs forth on the subject here: http://mail.python.org/pipermail/python-list/2003-May/205182.html This debate is almost six years old, but I don't think the basic issues have changed very much.