Do you think Unit Tests are a good way to show your fellow programmers how to use an API?

Question

Do you think Unit Tests are a good way to show your fellow programmers how to use an API?

I was listening to the Stackoverflow Podcast this week and I now realize that unit testing is not appropriate in all situations (I.E. it can cost you time if you go for 100% code-coverage). I agree with this as I have suffered from the "OCD code coverage disorder in the past), and have now mended my ways.

However to further appropriate my knowledge of the subject, I'd like to know if unit testing is a good way to bring in new programmers that are unfamiliar with the project's APIs. (It sure seems easier than just writing documentation...although I like it when there's documentation later...)

Answer 1

I think Unit testing is a fantastic way to document APIs. It doesn't necessarily replace good commenting or API docs but it is a very practical way to help people get into the nitty gritty of your code. Moreover good unit testing promotes good design, so chances are your code will be easier to understand as a result.

Answer 2

Unit testing, IMO, isn't a substitute for documentation in any way. You write tests to find and exercise the corner cases of your code, to make sure that all boundary conditions are met. These are usually not the most appropriate examples to give to someone who is trying to learn how the method works.

You also typically don't give as much explanation of why what's happening is happening in a unit test.

Finally, the contents of unit tests typically aren't as accessable to documentation generation tools, and are usually separated from the code they're testing (oddities like Python's doctests notwithstanding).

Answer 3

Good documentation usually includes good examples. It's hard for me to imagine a better set of examples than exactly the ones that show what's expected of a correct implementation!

In addition, maintenance is a crucial issue. It's good practice to deal with a defect by adding a test that exposes the defect, and then by making that test succeed without failing prior tests. If unit tests are regarded as part of the documentation, then this new test will be part of the documentation's change log, thus helping subsequent developers learn from the experience.

Answer 4

Although not a replacement, Unit Tests can be an excellent way of demonstrating the use of a class or API, particularly for library functions where the amount of code to perform tests is minimal.

For example out Math library is well unit-tested. If someone has a question (e.g. how to find information about ray/volume intersections) then the first-place I send them are the unit tests.

(Why do we test something that could be considered so stable? Well for one thing we support five platforms and gradually add platform-specific SIMD implementations based on benchmarking, for another the unit tests provide an excellent framework for adding new platforms or functionality).

FWIW I thought this weeks Podcasts discussion of unit testing was bang-on. Unit Testing (and Test Driven Development) is a great method for developing the components of your application, but for testing the application itself you quickly reach a point where the code needed for testing becomes disproportionate and brittle.

Answer 5

Documentation should not be substituted with unit test code, period.

Documentation is geared towards the programmer that is trying to use, and understand, your API.

Unit tests are geared towards covering corner cases, that although they should be documented, should not be the focus for a new user of your API.

Documentation should have a gradual build-up of complexity, unit tests should be as simple as possible yet as complex as necessary in order to test the functionality.

For instance, you might have many unit tests that look very alike, but have minute differences, just to cover various oddball corner cases and assert the correct behavior.

Rather than having the user of your API decipher these unit tests, figure out the differences, and figure out why this should produce different (or possibly the same) behavior is not a good teaching aid.

Unit tests are for maintainers of your api, the programmers that will fix bugs in it, or add new features to it, or refactor it, or ....

Documentation is for programmers that will use your api.

These are not the same target audiences.

One such subtle difference might be that while a unit test asserts that when a function gets passed a negative value, it will do something specific, the documentation would instead go into details about why that particular solution was picked. If all the documentation does is just rewrite the code into english, then there is not much point in it, so documentation usually is a lot more verbose and explanatory than the code.

Answer 6

No. Tests can be used as a secondary reference, they at least have the benefit of being examples that actually compile, but they are not a substitute for good API documentation.

Beware of zealots claiming almost mystical powers for unit tests saying things like "the tests are the design", "the tests are the specification", "the tests are the documentation". No. The tests are the tests and they don't give you an excuse to cut corners elsewhere.