What kinds of unit tests pay off the most in business value?

Question

My question assumes that folks already believe that unit tests of some sort are worthwhile and actually write them on their current projects. Let's also assume that unit tests for some parts of the code are not worth writing because they're testing trivial features. Examples are getters/setters, and/or things that a compiler/interpreter will catch immediately. The contrary assumption is that "interesting" code is worth testing.

Answer 1

The level of coverage for a region of code should be directly proportional to a combination of the likelihood of change and the amount of functionality that depends on it.

For example, we have some fairly complex parsing logic at the core of our base control classes. We unit-test the absolute piss out of it because it has to handle input which comes from systems outside our influence (high possibility of change) + ALL of our UI controls depend on it to be stable. The slightest tweak in the base ripples and magnifies through layers of inheritance and dependency.

Answer 2

The unit tests that verify data passed between layers often add the most value

UI <----> Business <---> Data

Answer 3

To take a shot at my own question, here's the modules I've found worthwhile testing:

a) Any of the business rules. On my current application the business rules are externalized from the business objects.

b) Any of the virtual attributes on the business objects. Often these virtual attributes have involved algorithms.

c) Major components where we can state the expected produced output for a given input consumed. Often this bleeds into integration testing.

d) Our development process requires us to check in unit tests for new features and bug fixes. For bug fixes, we attempt to duplicate exactly what caused the bug and show the unit test can pass once the code fix is checked in.

e) Any of our translation layers where we translate from one data representation type to another such as POJO-XML

Answer 4

I find that the tests with the most pay-off are the ones testing for previously found bugs. In my opinion, if a team were to do nothing but add regression tests for bugs as they fixed them, they would get more bang for their buck than almost any other testing strategy.

By testing what has been broken (and testing it thoroughly), you know that you're testing areas susceptible to breakage.

This of course assumes that you are testing things that break during development not just waiting for QA reports to trickle in after a deploy or something like that.

Answer 5

Discussed in SO Podcast 41 - this part is not in the transcript at the moment. The responses in the blog contain further discussion of unit test coverage value.

• Joel worries that excessive TDD (say, going from 90% to 100% test coverage) cuts time from other activities that could benefit the software product, such as better usability or additional features users are clamoring for.
• Unit tests are absolutely useful as a form of “eating your own dogfood”, and documenting the behavior of your system. Even if you disregard the actual results of the tests completely, they’re still valuable as documentation and giving you a fresh outside perspective on your codebase.
• Joel notes the difficulty of testing the UI (web or executable) versus testing the code behind the UI. The classic method of doing this is probably documented in The Art of UNIX Programming, where you start with a command-line app that takes in and spits out text. The GUI is simply a layer you paste on top of the command-line app, which leads to perfect testability — but perhaps not such great apps, in the long run. Which is more important?

Answer 6

As the question is asked, I have no doubt whatsoever that functional tests like the selenium test provides the greatest business value. A selenium (web) test clicks through the application and asserts the expected behavior, as defined by the user story/use cases. These tests assert that the main "thing" your software is trying to communicate to the customer is good.

If these things do not work, your customer will loose confidence in your ability to produce credible software, hence taking away a fair amount of business value.

Now there may be a fair amount of details that these tests do not cover, but I really believe they are essential in the creation of trust, especially when talking to a mass market.

Answer 7

All reasonable unit tests are valuable, but the ones that tend to save the most time in my experience, thus yielding the best ROI, are unit tests for bugfixes. Mostly because the bugfixes apply to the most difficult/brittle parts of the code.

Answer 8

TDD helps you write your code faster, so only TDD when going faster would provide more business value. ;)

Answer 9

Tests that can be reused in many places. Examples from my current project:

• syntax-checking of configuration files
• Checking that model properties referred to in RTF templates actually exist in the model classes (people change the classes and usually forget to adjust the templates)
• Checking of some oft-overlooked style guide rules (each UI mask must have a title bar text, each button must have a unique mnemonic, etc.)

Even programmers who are skeptical about unit tests love these because they can use them with a minimal amount of work - and in some cases it leads them towards writing unit tests of their own.