Unit testing infrastructure for a python module

Question

Hi there,

I'm writing a python module and I would like to unit test it. I am new to python and somewhat bamboozled by the options available.

Currently, I would like to write my tests as doctests as I like the declarative rather than imperative style (however, feel free to disabuse me of this preference if it is misinformed). This raises a few questions, however:

1. Where should I put the tests? In the same file as the code they are testing (or in docstrings for doctests)? Or is it considered better to separate them out into their own directory?
2. How can I run all the tests in the whole module from the command-line in one go?
3. How can I report the code coverage of the test suite?
4. Any other best-practices I should be aware of for unit testing in python?

Answer 1

feel free to disabuse me of this preference if it is misinformed

I believe I used doctest more extensively (way stretching its intended use boundaries) than any other open source developer, at least within a single project -- all the tests in my gmpy project are doctests. It was brand new at the time gmpy was starting, it seemed a great little trick, and if something is worth doing it's worth doing in excess -- right?-)

Wrong. Except for gmpy, where redoing everything as proper unit tests would be too much rework, I've never made that mistake again: these days, I use unit tests as unit tests, and doctests just to check my docs, as they've always been meant to be used. What doctests do (compare an expected with an actual result for equality -- that's all) is just not a good or sound basis to build a solid test suite on. It was never intended otherwise.

I would recommend you look at nose. The unittest module in the new Python 2.7 is much richer and nicer, and if you're stuck on 2.4, 2.5 or 2.6 you can still use the new features with the unittest2 which you can download and install; nose complements unittest quite well.

If you can't stand unittest (but -- give it a try, it grows on you!-), maybe try py.test, an alternative package with a pretty different philosophy.

But, please, don't stretch doctest to test stuff other than examples in docs! The exact-equality comparison will stand in your way far too often, as I've had to learn at my (metaphorical;-) expense in gmpy...

Answer 2

I don't like doctests for these reasons:

• You can't run a subset of the tests. When a test fails, it's useful to run just one test. Doctest provides no way to do that.
• If a failure happens in the middle of the doctest, the whole thing stops. I'd rather see all the results to decide how to tackle a breakage.
• The coding style is stylized, and has to have printable results.
• Your code is executed in a special way, so it's harder to reason about how it will be executed, harder to add helpers, and harder to program around the tests.

This list was taken from my blog post Things I don't like about doctest, where there's more, and a long thread of comments debating the points.

About coverage: I don't believe there's a coverage tool for Python that will measure coverage within doctests. But since they are simply long lists of statements with no branches or loops, is that a problem?

Answer 3

For coverage, check out the excellent coverage.py.

Otherwise, everything Alex Martelli wrote is very much on point.

Answer 4

doctests are great for quick, minor unit tests that describe what some of the basic usages of the objects in question, (as they show up in docstrings, and hence help(whatever), etc).

I've personally found extensive and more thorough testing to be more effective using the unittest module, and now the 2.7 module (back ported to unittest2) has even more handy assertions. You can set up test suites and as complex a scenario as you like with the unit testing framework and cover whole swaths of different tests in one go (command-line wise)

coverage.py, by Ned Batchelder and as @bstpierre mentions will work with either of these, and I recommend it for seeing what you've got tested of the code and what doesn't. You can add it into a CI system (i.e. Hudson or whatever you like to use) to keep up on what's covered and not, and the HTML reports are fantastic for seeing what hasn't been hit with testing coverage. Coverage supports Junit xml output, which many CI systems know how to provide charted on-going results to let you see the build getting better or worse over time.

Answer 5

I have this suspicion that Alex might be a fair bit ahead of me on the programmer's curve, but if you want the perspective of somebody with some Python experience (as a "user" rather than an expert or evangelist), yet not in the same league, my findings about unit testing have been pretty much the same.

Doctests might sound great for simple testing in the beginning, and I went in that direction for some personal project at home because it had been recommended elsewhere. At work we use nose (although so canned and wrapped up I was under the impression we'd been using pyUnit until not long ago), and a few months back I moved to nose at home too.

The initial setup time and management overhead, and the separation from the actual code, might seem unnecessary in the beginning, especially when you're testing something that isn't that large a codebase, but in the long run I've found doctests getting in the way of every single refactoring or restructuring I wanted to do, rather hard to maintain, practically impossible to scale, and offsetting the initial savings very quickly. And yes, I'm aware that unit testing isn't the same as integration testing, but doctests tend to define your units for you rather too strictly. They're also not well suited to unit based agile if you ever decide it's a valid sketching tool or dev model.

It might take you a bit to plan and then refine your unit tests the way pyUnit or nose steer you towards, but chances are that even in the short term you'll find it's actually helping you out on many levels. I know it did for me, and I'm relatively new to the complexity and scale of the codebase I'm working on these days. Just have to clench your teeth for the first few weeks.

Answer 6

I agree with all the above points raised about doctest not scaling and I prefer to stick with unittest.

One tip I can contribute is to invoke the unit tests from the code handling __name__ == "__main__ so if the test file is run as a script it will run its tests.

eg:

#!/usr/bin/env python

"""
Unit tests for the GetFiles.py utility
"""

import unittest
from FileUtilities import getTree

class TestFileUtilities(unittest.TestCase):

   def testGetTree(self):
      """
      Tests that a known tree is found and incidentally confirms
      that we have the tree we expected to use for our current
      sample extraction.
      """
      found = getTree('./anzmeta-dtd', '.pen')
      expected_path_tail = ['ISOdia.pen',
                       'ISOgrk1.pen',
                       'ISOtech.pen']
      for i, full_path in enumerate(found):
         assert full_path.endswith( expected_path_tail[i] ), expected_path_tail[i]

# other tests elided         

if __name__ == "__main__":
   # When this module is executed from the command-line, run all its tests
   unittest.main()