What is your strategy to avoid dynamic typing errors in Python (NoneType has no attribute x)?

Question

Python is one of my favorite languages, but I really have a love/hate relationship with it's dynamicness. Apart from the advantages, it often results in me forgetting to check a type, trying to call an attribute and getting the NoneType (or any other) has no attribute x error. A lot of them are pretty harmless but if not handled correctly they can bring down your entire app/process/etc. Over time I got better predicting where these could pop up and adding explicit type checking, but because I'm only human I miss one occasionally and then some end-user finds it.

So I'm interested in your strategy to avoid these. Do you use type-checking decorators? Maybe special object wrappers? Please share...

Answer 1

If you write good unit tests for all of your code, you should find the errors very quickly when testing code.

Answer 2

I tend to use

if x is None:
    raise ValueError('x cannot be None')

But this will only work with the actual None value.

A more general approach is to test for the necessary attributes before you try to use them. For example:

def write_data(f):
    # Here we expect f is a file-like object.  But what if it's not?
    if not hasattr(f, 'write'):
        raise ValueError('write_data requires a file-like object')
    # Now we can do stuff with f that assumes it is a file-like object

The point of this code is that instead of getting an error message like "NoneType has no attribute write", you get "write_data requires a file-like object". The actual bug isn't in write_data(), and isn't really a problem with NoneType at all. The actual bug is in the code that calls write_data(). The key is to communicate that information as directly as possible.

Answer 3

I haven’t done a lot of Python programming, but I’ve done no programming at all in staticly typed languages, so I don’t tend to think about things in terms of variable types. That might explain why I haven’t come across this problem much. (Although the small amount of Python programming I’ve done might explain that too.)

I do enjoy Python 3’s revised handling of strings (i.e. all strings are unicode, everything else is just a stream of bytes), because in Python 2 you might not notice TypeErrors until dealing with unusual real world string values.

Answer 4

Something you can use to simplify your code is using the Null Object Design Pattern (to which I was introduced in Python Cookbook).

Roughly, the goal with Null objects is to provide an 'intelligent' replacement for the often used primitive data type None in Python or Null (or Null pointers) in other languages. These are used for many purposes including the important case where one member of some group of otherwise similar elements is special for whatever reason. Most often this results in conditional statements to distinguish between ordinary elements and the primitive Null value.

This object just eats the lack of attribute error, and you can avoid checking for their existence.

It's nothing more than

class Null(object):

    def __init__(self, *args, **kwargs):
        "Ignore parameters."
        return None

    def __call__(self, *args, **kwargs):
        "Ignore method calls."
        return self

    def __getattr__(self, mname):
        "Ignore attribute requests."
        return self

    def __setattr__(self, name, value):
        "Ignore attribute setting."
        return self

    def __delattr__(self, name):
        "Ignore deleting attributes."
        return self

    def __repr__(self):
        "Return a string representation."
        return "<Null>"

    def __str__(self):
        "Convert to a string and return it."
        return "Null"

With this, if you do Null("any", "params", "you", "want").attribute_that_doesnt_exists() it won't explode, but just silently become the equivalent of pass.

Normally you'd do something like

if obj.attr:
    obj.attr()

With this, you just do:

obj.attr()

and forget about it. Beware that extensive use of the Null object can potentially hide bugs in your code.

Answer 5

One advantage of TDD is that you end up writing code that is easier to write tests for.

Writing code first and then the tests can result in code that superficially works the same, but is much harder to write 100% coverage tests for.

Each case is likely to be different

It might make sense to have a decorator to check whether a particular parameter is None (or some other unexpected value) if you use it in a bunch of places.

Maybe it is appropriate to use the Null pattern - if the code is blowing up because you are setting the initial value to None, you could instead set the initial value to a null version of the object.

More and more wrappers can add up to quite a performance hit though, so it's always better to write code from the start that avoids the corner cases

Answer 6

forgetting to check a type

This doesn't make much sense. You so rarely need to "check" a type. You simply run unit tests and if you've provided the wrong type object, things fail. You never need to "check" much, in my experience.

trying to call an attribute and getting the NoneType (or any other) has no attribute x error.

Unexpected None is a plain-old bug. 80% of the time, I omitted the return. Unit tests always reveal these.

Of those that remain, 80% of the time, they're plain old bugs due to an "early exit" which returns None because someone wrote an incomplete return statement. These if foo: return structures are easy to detect with unit tests. In some cases, they should have been if foo: return somethingMeaningful, and in still other cases, they should have been if foo: raise Exception("Foo").

The rest are dumb mistakes misreading the API's. Generally, mutator functions don't return anything. Sometimes I forget. Unit tests find these quickly, since basically, nothing works right.

That covers the "unexpected None" cases pretty solidly. Easy to unit test for. Most of the mistakes involve fairly trivial-to-write tests for some pretty obvious species of mistakes: wrong return; failure to raise an exception.

Other "has no attribute X" errors are really wild mistakes where a totally wrong type was used. That's either really wrong assignment statements or really wrong function (or method) calls. They always fail elaborately during unit testing, requiring very little effort to fix.

A lot of them are pretty harmless but if not handled correctly they can bring down your entire app/process/etc.

Um... Harmless? If it's a bug, I pray that it brings down my entire app as quickly as possible so I can find it. A bug that doesn't crash my app is the most horrible situation imaginable. "Harmless" isn't a word I'd use for a bug that fails to crash my app.

Answer 7

One tool to try to help you keep your pieces fitting together well is interfaces. zope.interface is the most notable package in the Python world for using interfaces. Check out http://wiki.zope.org/zope3/WhatAreInterfaces and http://glyph.twistedmatrix.com/2009/02/explaining-why-interfaces-are-great.html to start to get an idea how interfaces and z.i in particular work. Interfaces can prove very useful in a large Python codebases.

Interfaces are no substitute for testing. Reasonably comprehensive testing is especially important in highly dynamic languages like Python where there are types of bugs that could not exist in a statically types language. Tests will also help you catch the sorts of bugs that are not unique to dynamic languages. Fortunately, developing in Python means that testing is easy (due to the flexibility) and you have plenty of time to write them that you saved because you're using Python.

Answer 8

You can also use decorators to enforce the type of attributes.

>>> @accepts(int, int, int)
... @returns(float)
... def average(x, y, z):
...     return (x + y + z) / 2
...
>>> average(5.5, 10, 15.0)
TypeWarning:  'average' method accepts (int, int, int), but was given
(float, int, float)
15.25
>>> average(5, 10, 15)
TypeWarning:  'average' method returns (float), but result is (int)
15

I'm not really a fan of them, but I can see their usefulness.

Answer 9

forgetting to check a type

With duck typing, it shouldn't be necessary to check a type. But that's theory, in reality you will often want to validate input parameters (e.g. checking a UUID with a regex). For that purpose, I created myself some handy decorators for simple type and return type checking which are called like this:

@decorators.params(0, int, 2, str) # first parameter must be integer / third a string
@decorators.returnsOrNone(int, long) # must return an int/long value or None
def doSomething(integerParam, noMatterWhatParam, stringParam):
    ...

For everything else I mostly use assertions. Of course one often forgets to check a parameter, so it's necessary to test and to test often.

trying to call an attribute

Happens to me very seldom. Actually I often use methods instead of direct access to attributes (the "good" old getter/setter approach sometimes).

because I'm only human I miss one occasionally and then some end-user finds it

"Software is always completed at the customers'." - An anti-pattern which you should solve with unit tests that handle all possible cases in a function. Easier said than done, but it helps...

As for other common Python mistakes (mistyped names, wrong imports, ...), I'm using Eclipse with PyDev for projects (not for small scripts). PyDev warns you about most of the simple kinds of mistakes.