Are Tests Valid Documentation?

Question

As suggested, I have closed the question posed here and have broken it up into separate questions regarding code documentation to be posed throughout the day.

---

The first part: How do you document your methods?

This question is about documentation as it relates to automated tests, especially unit tests.

Do you think it is appropriate to use well written unit tests as a replacement for documentation? When starting in on an unfamiliar piece of code do you go first to the tests or the doc comments?

On the flip side, when does a test need its own documentation? Is it ever acceptable to have a test so complicated? What do you do to minimize the friction? What are the guidelines that you follow and would like to see others follow as well?

As before, do you find that your opinions differ depending on project size/type? On language you're using? On tools available?

Answer 1

For my two cents, I think it is occasionally appropriate to use tests as replacement for doc comments on simple and well named code and only on code that you never intend to release as a closed API. Sometimes I even prefer it, as especially comments written by non-native English speakers can be tougher to get through than the code.

As for unit test documentation, my fixtures almost never get documentation. If I think that what is happening within the test itself is tricky such as if it is initializing the object being tested in an unconventional way, or if I feel the method name doesn't do the test justice I will write out a short list of what happens and what my expectations are. I do not shy away from long method names however so this is not frequently the case. I also have two types of naming styles for my tests. My unit tests are typical ComponentOneTests...WhenInputIsNullThrowsError style. For anything where I'm trying to test component interaction I tend to default to BDD (WhenInputIsValid...SendInputToComponent1AndRelayResultToComponent2).

Answer 2

I think tests can contribute quite a bit to your documentation, but they aren't sufficient in and of themselves. There's often a lot of noise in unit tests (i.e. mocking), and sometimes tests do things outside of the normal band in an attempt to simplify tests or defeat friction. In a perfect BDD world, you would have a collection of specifications that represent both your requirements and your test cases. Unfortunately, in most projects (particularly those that carry a large legacy weight), that's not feasible.

Doc comments are very useful when working with languages that support IDEs that can read and display the comments in auto-completion. However, they're only really useful to describe the individual puzzle pieces -- they don't really explain how the puzzle goes together. That's where "real" documentation comes in (although, again, unit tests can help in this regard as well).

In my opinion, tests should always be self-documenting, at least in terms of "why is this behavior being tested". Sometimes you might have to comment the "how", though, if your test does something particularly exotic.

Answer 3

Tests are valid documentation, but most of them would not be what I consider sufficient to be the only documentation. As much as anything, perhaps more, documentation has to be sculptured to the audience that is going to consume it.

If you ask if a well written test is sufficient for an internal method? Well probably.

Is it ok for a public call into an API? Almost certainly not, could you imagine only having a unit test for some of the .Net objects?

For a consumer feature? I don't know about you, but if I was using TurboTax, I asked for help, and got a Unit Test back, there would be a fair bit of cursing going on via their Hotline.

So how about "It depends".

Answer 4

I don't think they can fully replace any kind of documentation, but they can be very handy as a sort of documentation-by-example. So, if you're wondering "What happens if I pass null to this method?" a unit test can give you a better answer than comments written once and never updated.

However, since unit tests are often split up into small chunks to limit the number of assertions each one has, they might not be handy to get a general overview of a complex method, and even less so at a class or module level.

On the flip side, when does a test need its own documentation? Is it ever acceptable to have a test so complicated?

Tests should be as simple as possible, definitely avoiding conditional logic and such. Initialisation can get complex, so an Object Mother might come in handy, or even better, refactor so you can get your stubs/mocks in earlier.

Answer 5

I've been using the AAA style of testing which is a method of BDD.

With the addition of namespaces you can get great descriptive sentences for your system embedded within your code.

I've followed a pattern of [System].[Section].[Context].[Action].[Assertion] eg:

namespace AccountingProgram.Transfers.WhenManagingAccounts
{

    [TestFixture]
    public class IfATransferIsAttemptedForAnAmountGreaterThanTheCustomersBalance
    {
        ... Arranging (aka. Setup) ...

        ... Act (the actual method call) ...

        [Test]
        public void TheTransactionIsDenied()
        {
            someMockClass.AssertSomeMethodWasntCalled();
        }

        ... Other assertions ...

    }
}

As others have said, on their own they probably don't give enough depth, but they can be a valuable addition or starting point for your documentation.

Answer 6

No, well written unit tests can help supplement documentation by giving concrete examples but the unit tests are rarely comprehensive and manageable, e.g. even for a simple temperature conversion program there are millions of possible inputs that one would have to do to cover everything and that wouldn't be easily manageable from my view. I usually go to the doc comments as the tests may or may not be there as Web UI automated tests tend to be rare I've found.

Every test should have the documentation of why is this being done, what are the inputs and what is the expected result. I suppose some may not call this documentation but rather just part of the bookkeeping involved with testing. There can be cases where there is complex business logic and so the why for a test could be to note those rules. I've seen a variety of different techniques for tracking tests from scripts to excel spreadsheets and think that sometimes simple methods can work though it is good for an organization to be consistent with how it wants to handle this part of a project.

Yes, I've had a wide variety of opinions depending on project size and technologies involved as well as the manager's preference as some like seeing testing documented and others couldn't care less if it exists at all. There is something to be said for personal preference and timing as sometimes there is time to well document things and sometimes that gets cut out has been my experience.

JB