A good way to write unit tests

Question

So, I previously wasn't really in the practice of writing unit tests - now I kind of am and I need to check if I'm on the right track.

Say you have a class that deals with math computations.

class Vector3
{
public:  // Yes, public.
  float x,y,z ;
  // ... ctors ...
} ;

Vector3 operator+( const Vector3& a, const Vector3 &b )
{
  return Vector3( a.x + b.y /* oops!! hence the need for unit testing.. */,
                  a.y + b.y,
                  a.z + b.z ) ;
}

There are 2 ways I can really think of to do a unit test on a Vector class:

1) Hand-solve some problems, then hard code the numbers into the unit test and pass only if equal to your hand and hard-coded result

bool UnitTest_ClassVector3_operatorPlus()
{
  Vector3 a( 2, 3, 4 ) ;
  Vector3 b( 5, 6, 7 ) ;

  Vector3 result = a + b ;

  // "expected" is computed outside of computer, and
  // hard coded here.  For more complicated operations like
  // arbitrary axis rotation this takes a bit of paperwork,
  // but only the final result will ever be entered here.
  Vector3 expected( 7, 9, 11 ) ;

  if( result.isNear( expected ) )
    return PASS ;
  else
    return FAIL ;
}

2) Rewrite the computation code very carefully inside the unit test.

bool UnitTest_ClassVector3_operatorPlus()
{
  Vector3 a( 2, 3, 4 ) ;
  Vector3 b( 5, 6, 7 ) ;

  Vector3 result = a + b ;

  // "expected" is computed HERE.  This
  // means all you've done is coded the
  // same thing twice, hopefully not having
  // repeated the same mistake again
  Vector3 expected( 2 + 5, 6 + 3, 4 + 7 ) ;

  if( result.isNear( expected ) )
    return PASS ;
  else
    return FAIL ;
}

Or is there another way to do something like this?

Answer 1

I believe that writing out the numbers (your second approach) is the correct option. It makes your intent much more obvious to someone reading the test.

Suppose you weren't overloading the + operator, but instead had a horribly named function f that took two Vector3s. You didn't document it either, so I looked at your tests to see what f was supposed to do.

If I see Vector3 expected( 7, 9, 11 ), I have to go back and reverse-engineer how 7, 9, and 11 were the "expected" results. But if I see Vector3 expected( 2 + 5, 6 + 3, 4 + 7 ), then it is clear to me that f adds the individual elements of the arguments into a new Vector3.

---

You didn't ask this in your question, but I'd like to make another point on the side. As for which tests to write, you really want to make sure you cover edge cases as well. What should happen for

Vector3 a(INT_MAX, INT_MAX, INT_MAX);
Vector3 b(INT_MAX, INT_MAX, INT_MAX);

Vector3 result = a + b;

// What is expected?  Simple overflow?  Exception?  Default to invalid value?

If you were doing division, you'd make sure to cover the division by zero case. Try to keep those kinds of edge cases in mind.

Answer 2

Way #1 is the generally accepted way of doing Unit testing. By rewriting your code, you could be rewriting faulty code into the test. A lot of the time, only one real test case is needed per method that you're testing, so it's not TOO time consuming.

Answer 3

Duplicating that logic won't really help a whole lot. You understand that reading your comments on #2 :). Unless it's something incredibly complicated, I would use method #1.

It might take a little bit of work upfront to determine some test data; but this is usually pretty easy to determine.

Answer 4

It depends always on the use case. I would choose always that version, that makes the tested idea more obvious. For that reason I would also not use the isNear method. I would check for

expected.x == 7; expected.y == 9; expected.z == 11;

Using a good xUnit library you will get a clean error message which component of expected was wrong. In your example you would have to search for the real source of the error.

Answer 5

Way 1 would be the better option. The main thing is how you chose the magic data against which code will be tested.

The other way can be, Some time instead of hard coding values into unit test, We can have input set( Magic data ) and a set of expected outcomes respective to input. So unit test will read value from input set, execute code and test against expected outcome.

Answer 6

It would be completely pointless to use the same calculation in the test as in the code. If you're going to be extra careful, why not be extra careful when actually writing the code? Using examples calculated by hand is the best way to do it, but even better would be to write the test before you write the code, that way you can't be lazy and write a test you know will pass and avoid the edge cases you're not completely sure about.

Answer 7

With vector addition, it doesn't really matter which method you choose, because it's a pretty simple operation. A better illustration might be testing say, a normalize method:

Vector3 a(7, 9, 11); 
Vector3 result = a.normalize(); 

Vector3 hand_solved(0.4418, 0.5680, 0.6943);
Vector3 reproduced(7/sqrt(7*7+9*9+11*11), 9/sqrt(7*7+9*9+11*11), 
    11/sqrt(7*7+9*9+11*11));

See? It's not clear to a reader that either is correct. The reproduced calculation is verifiable, but it's messy and hard to read. It's also not practical to rewrite every computation in a unit test. The hand-solved calculation doesn't offer any assurances to the reader that it is correct (the reader would have to solve by hand and compare answers).

The solution is to pick simpler inputs. With vectors, you can test all operations on just the basis vectors (i, j, k). So in this particular case it would be clearer to say something like:

Vector3 i(1, 0, 0);
Vector3 result = i.normalize();
Vector3 expected(1, 0, 0);

Here it's clear what you are testing and what you expect the result to be. If the reader knows what normalize is supposed to do, then it will be clear that the answer is correct.

Answer 8

My approach to this is fairly simple: Never ape the production code to get your result in the test. If your algorithm is flawed, then your unit test is both reproducing the flaw and passing. Take a second to think about it! Bugged code and a bugged, passing test. I don't think it can get any worse. Imagine you find the bug in your code and change it; the test will now fail, but it looks correct. IMO not only does it make for error prone testing, but it makes you think about the result in terms of the algorithm. For things like maths, you shouldn't care what the algorithm is, merely that the answer is correct. I'd go as far as to say that I fundamentally distrust tests that ape the production code's logic.

Tests should be as declarative as possible, and that means hard coding the fully calculated result. For maths tests, I generally work out the result on paper / a calculator using values that are as simple as possible, but no simpler. E.g. if I wanted to test a normalize method, I'd pick some well known values. Most people know that sin/cos of 45 is one over root two, so normalizing (1,-1, 0) will give an easily recognisable value. There are numerous other well-known numbers / tricks you can use. You can encode your results using well-named constants to aid readability, too.

I also recommend using data-driven testing for maths types, as you can rapidly add new test cases.

Answer 9

Hi,

You should do number 1 anyway to verify your code is correct - the unit test should have been done hypothetically as part of creating the calculation. Using this knowledge you can create your unit test to use the code you have already created (ie, don't duplicate it).

The unit test should test known success cases, known failure cases, boundary cases (upper/lower ranges if applicable) and any rare cases (rare and expensive to debug at runtime, but very inexpensive to test at build time, assuming you know what they are :)

You will find straight calculations are the easiest to unit test as the flow of the logic is (hopefully) self contained.

Answer 10

Simple rules to follow:

1. Always use Arrange, Act and Assert (AAA pattern) - google to find more about this.
2. you should NEVER have if/else blocks in your unit tests
3. you should NEVER have any calculation/logic in your unit tests

4. Don't test more than one thing in you unit test. For example: If I have written a method : public int Sum(int number1, int number2) I will have 4-5 unit tests, which will look like this

   Test_Sum_Number1IsOneNumer2IsTwo_ReturnsThree

   Test_Sum_Number1IsZeroNumer2IsZero_Returns0

   Test_Sum_Number1IsNegativeOneNumer2IsNegativeThree_ReturnsNegativeFour ....so on and so forth

Or maybe instead of writing four different methods you can use RowTest attribute in MBUnit or TestCase in NUnit(2.5.5 onwards) to have parametrized tests - here you just write one method and pass in different parameters by specifying them as attributes.