Does Python doctest remove the need for unit-tests?

Question

Hi all,

A fellow developer on a project I am on believes that doctests are as good as unit-tests, and that if a piece of code is doctested, it does not need to be unit-tested. I do not believe this to be the case. Can anyone provide some solid, ideally cited, examples either for or against the argument that doctests replace the need for unit-tests?

Thank you -Daniel

EDIT: Can anyone provide a reference showing that doctesting should not replace unit-testing?

Answer 1

It is possible to implement a complete unit test suite with doctests. However, doctests are a bit limited and it's usually better to reserve that for simple documentation examples, and use a more robust unit test framework (eg. unittest) for the real, detailed unit tests.

Another advantage of unittest is that the test framework will be familiar to somebody coming from a different development environment that uses JUnit, NUnit, or similar. The doctest module is a little different.

Answer 2

There really is no arguments for or against doc tests. They are simply tests. In fact as of python 2.4 there is a way to create unit test suits from your doctests: http://docs.python.org/library/doctest.html#unittest-api

In a sense you can think of doc-tests as a subset of unit tests, at least from the functional point of view.

However python docs suggest using doctests for things like:

• examples in documentation
• regression testing
• writing tutorial documentation

Their argument is that if you write your tests within the documentation you will be forced to write both better docs and tests. As opposed to when you write unit tests separately - you rarely add sufficient documentation, and they tend to stagnate and go out of date.

See: http://docs.python.org/library/doctest.html#soapbox

I imagine doctests would be harder to write for things like integration testing. However, as you can see on python and Django - they make extensive use of doctests, which yields much more understandable documentation and tutorials.

It all depends on your project. There is no "right" way to test.

Also I recommend taking a look at Code Complete 2 book, as you may find that unit-testing is not the best way to make sure your software is fault free.

Answer 3

I (ab)used doctest in lieu of unittest, back when I started my gmpy project many years ago -- you can browse its sources and see that all the functionality is thoroughly tested with doctests (the functionality's supplied by a C-coded Python extension, and last time I instrumented it for coverage measurement I was over 95% coverage). Why did I do that? Because doctest was brand new, as was gmpy, and I was curious to see how far I could push it.

Answer: very far indeed -- but definitely not worth it (the novelty wears off, but you don't want to rewrite all your tests, which is why gmpy's tests are still all-doctests). The extreme fragility of doctests, where even the tiniest typo fix in a message breaks the test, is a real bother when they're being abused this way. It's kind of like traditional integration tests based on comparing output with a "golden" (known-good) expected output: easy to write the first time around, but you'll repent at leisure after a few years of fixing gratuitous test breakages;-).

If you find unittest's style onerous, there are other excellent alternatives that are still meant for use in unit tests, such as py.test and nose -- doctest is really meant for a different purpose (supporting docs, not generic unit tests) and while it's of course worth adding whatever doctests you've written for docs purposes to your test battery, it's not worth the test-maintenance headaches of replacing unit tests with it.

Answer 4

There's a concrete example in the Python standard library that persuades me that doctests alone aren't always enough, namely the decimal module. It has over 60000 individual testcases (in Lib/test/decimaltestdata); if all those were rewritten as doctests, the decimal module would become very unwieldy indeed. It's possible the number of tests could be slimmed down whilst still giving good coverage, but many of the numerical algorithms are sufficiently complicated that you need huge numbers of individual tests to cover all possible combinations of branches.

Answer 5

doctests are great for some uses

• working and up to date documentation
• sample tests embeded in docstrings
• spikes or design phases when classes API is not really clear

unit tests are better in differents cases:

• when you need clear and somewhat complex setup/teardown
• when trying to get better coverage of all cases, inclusinf corner cases
• for keeping tests independant from each other

In other words, at least for my own use, doctests are great when you focus on explaining what you are doing (docs, but also design phases) but more of a burden when you intent to use tests as a seat belt for refactoring or code coverage.