Writing standards for unit testing

Question

I plan to introduce a set of standards for writing unit tests into my team. But what to include?

These two posts (Unit test naming best practices and Best practices for file system dependencies in unit/integration tests) have given me some food for thought already.

Other domains that should be covered in my standards should be how test classes are set up and how to organize them. For example if you have class called OrderLineProcessor there should be a test class called OrderLineProcessorTest. If there's a method called Process() on that class then there should be a test called ProcessTest (maybe more to test different states).

Any other things to include?

Does your company have standards for unit testing?

EDIT: I'm using Visual Studio Team System 2008 and I develop in C#.Net

Answer 1

You should probably take a look at the "Pragmatic Unit Testing" series. This is the C# version but there is another for Java.

With respect to your spec, I would not go overboard. You have a very good start there - the naming conventions are very important. We also require that the directory structure match the original project. Coverage also needs to extend to boundary cases and illegal values (checking for exceptions). This is obvious but your spec is the place to write it down for that argument that you'll inevitably have in the future with the guy who doesn't want to test for someone passing an illegal value. But don't make the spec more than a few pages or no one will use it for a task that is so context-dependent.

Update: I disagree with Mr. Potato Head about only one assert per Unit Test. It sounds quite fine in theory but, in practice, it leads to either loads of mostly redundant tests or people doing tons of work in setup and tear-down that itself should be tested.

Answer 2

If you are using tools from the family of Junit (OCunit, SHunit, ...), names of tests already follow some rules.

For my tests, I use custom Doxygen tags in order to gather their documentation in a specific page.

Answer 3

1. Try to use as few assert statements per test method as possible. This makes sure that the purpose of the test is well-defined.
2. I know this will be controversial, but don't test the compiler - time spent testing Java Bean accessors and mutators is better spent writing other tests.
3. Try, where possible, to use TDD instead of writing your tests after your code.

Answer 4

I've found that most testing conventions can be enforced through the use of a standard base class for all your tests. Forcing the tester to override methods so that they all have the same name.

I also advocate the Arrange-Act-Assert (AAA) style of testing as you can then generate fairly useful documentation from your tests. It also forces you to consider what behaviour you are expecting due to the naming style.

Answer 5

Another item you can put in your standards is to try and keep your unit test size small. That is the actuall test methods themselves. Unless you are doing a full integration unit test there usually is no need for large unit tests, like say more than 100 lines. I'll give you that much in case you have a lot of setup to get to your one test. However if you do you should maybe refactor it.

People also talk about refactoring there code make sure people realize that unit tests is code too. So refactor, refactor, refactor.

I find the biggest problem in the uses I have seen is that people do not tend to recognize that you want to keep your unit tests light and agile. You don't want a monolithic beast for your tests after all. With that in mind if you have a method you are trying to test you should not test every possible path in one unit test. You should have multiple unit tests to account for every possible path through the method.

Yes if you are doing your unit tests correctly you should on average have more lines of unit test code than your application. While this sounds like a lot of work it will save you alot of time in the end when comes time for the inevitable business requirement change.

Answer 6

Users of full-featured IDE's will find that "some of them" have quite detailed support for creating tests in a specific pattern. Given this class:

public class MyService {
    public String method1(){
        return "";
    }

    public void method2(){

    }

    public void method3HasAlongName(){

    }
}

When I press ctrl-shift-T in intellij IDEA I get this test class after answering 1 dialog box:

public class MyServiceTest {
    @Test
    public void testMethod1() {
        // Add your code here
    }

    @Test
    public void testMethod2() {
        // Add your code here
    }

    @Test
    public void testMethod3HasAlongName() {
        // Add your code here
    }
}

So you may want to take a close look at tool support before writing your standards.

Answer 7

Have a look at Michael Feathers on what is a unit test (or what makes unit tests bad unit tests)

Have a look at the idea of "Arrange, Act, Assert", i.e. the idea that a test does only three things, in a fixed order:

• Arrange any input data and processing classes needed for the test
• Perform the action under test
• Test the results with or more asserts. Yes, it can be more than one assert, so long as they all work to test the action that was performed.

Have a Look at Behaviour Driven Development for a way to align test cases with requirements.

Also, my opinion of standard documents today is that you shouldn't write them unless you have to - there are lots of resources available already written. Link to them rather than rehashing their content. Provide a reading list for developers who want to know more.

Answer 8

I use nearly plain English for my unit test function names. Helps to define what they do exactly:

TEST( TestThatVariableFooDoesNotOverflowWhenCalledRecursively )
{
/* do test */
}

I use C++ but the naming convention can be used anywhere.

Answer 9

I follow the BDD style of TDD. See: http://blog.daveastels.com/files/BDD_Intro.pdf http://dannorth.net/introducing-bdd http://behaviour-driven.org/Introduction

In short this means that

• The tests are not thought as "tests", but as specifications of the system's behaviour (hereafter called "specs"). The intention of the specs is not to verify that the system works under every circumstance. Their intention is to specify the behaviour and to drive the design of the system.

• The spec method names are written as full English sentences. For example the specs for a ball could include "the ball is round" and "when the ball hits a floor then it bounces".

• There is no forced 1:1 relation between the production classes and the spec classes (and generating a test method for every production method would be insane). Instead there is a 1:1 relation between the behaviour of the system and the specs.

Some time ago I wrote TDD tutorial (where you begin writing a Tetris game using the provided tests) which shows this style of writing tests as specs. You can download it from http://www.orfjackal.net/tdd-tutorial/tdd-tutorial_2008-09-04.zip The instructions about how to do TDD/BDD are still missing from that tutorial, but the example code is ready, so you can see how the tests are organized and write code that passes them.

You will notice that in this tutorial the production classes are named such as Board, Block, Piece and Tetrominoe which are centered around the concepts of a Tetris game. But the test classes are centered around the behaviour of the Tetris game: FallingBlocksTest, RotatingPiecesOfBlocksTest, RotatingTetrominoesTest, FallingPiecesTest, MovingAFallingPieceTest, RotatingAFallingPieceTest etc.

Answer 10

Make sure to include what is not an unit tests. See: http://stackoverflow.com/questions/109432/what-not-to-test-when-it-comes-to-unit-testing/110403#110403

Include a guideline so integration tests are clearly identified and can be run separately from unit tests. This is important, because you can end with a set of "unit" tests that are really slow if the unit tests are mixed with other types of tests.

Check this for more info on it: http://stackoverflow.com/questions/589603/how-can-i-improve-my-junit-tests/589620#589620 ... specially the second update.