Should a Unit-test replicate functionality or Test output?

Question

I've run into this dilemma several times. Should my unit-tests duplicate the functionality of the method they are testing to verify its integrity? OR Should unit tests strive to test the method with numerous manually created instances of inputs and expected outputs?

I'm mainly asking the question for situations where the method you are testing is reasonably simple and its proper operation can be verified by glancing at the code for a minute.

Simplified example (in ruby):

def concat_strings(str1, str2)
  return str1 + " AND " + str2
end

Simplified functionality-replicating test for the above method:

def test_concat_strings
  10.times do
    str1 = random_string_generator
    str2 = random_string_generator
    assert_equal (str1 + " AND " + str2), concat_strings(str1, str2)
  end
end

I understand that most times the method you are testing won't be simple enough to justify doing it this way. But my question remains; is this a valid methodology in some circumstances (why or why not)?

Answer 1

In unit testing you should definitely manually come up with test cases (so input, output and what side-effects you are expecting - these will be expectations on your mock objects). You come up with these test cases in a way so that they cover all functionality of your class (e.g. all methods are covered, all branches of all if statements, etc.). Think about it more along the lines of creating documentation of your class by showing all possible usages.

Reimplementing the class is not a good idea, because not only you get obvious code/functionality duplication, but also it is likely that you will introduce the same bugs in this new implementation.

Answer 2

to test the functionality of a method i'd use input and output pairs wherever possible. otherwise you might be copy&pasting the functionality as well as the errors in its implementation. what are you testing then? you would be testing if the functionality (including all of its errors) hasn't changed over time. but you wouldn't be testing the correctness of the implementation.

testing if the functionality hasn't changed over time might (temporarily) be useful during refactoring. but how often do you refactor such small methods?

also unit tests can be seen as documentation and as specification of a method's inputs and expected outputs. both should be as simple as possible so others can easily read and comprehend it. as soon as you introduce additional code/logic into a test it becomes harder to read.

your test actually looks like a fuzz test. fuzz tests can be very useful, but in unit tests randomness should be avoided due to reproducibility.

Answer 3

Testing the functionality by using the same implementation, doesn't test anything. If one has a bug in it, the other will as well.

But testing by comparing with an alternative implementation is a valid approach. For example, you might test a iterative (fast) method of calculating fibonacci numbers by comparing it with a trivial recursive, yet slow implementation of the same method.

A variation of this is using an implementation, that only works for special cases. Of course in that case you can use it only for such special cases.

When choosing input values, using random values most of the time isn't very effective. I'd prefer carefully chosen values anytime. In the example you gave, null values and extremely long values which won't fit into a String when concatenated come to mind.

If you use random values, make sure, you have a way to recreate the exact run, with the same random values, for example by logging the seed value, and having a way to set that value at start time.

Answer 4

A Unit-Test should exercise your code, not something as part of the language you are using.

If the code's logic is to concatenate strings in a special way, you should be testing for that - otherwise you need to rely on your language/framework.

Finally, you should create your unit tests to fail first "with meaning". In other words, random values shouldn't be used (unless you're testing your random number generator isn't returning the same set of random values!)

Answer 5

It's a controversial stance, but I believe that unit testing using Derived Values is far superior to using arbitrary hard-coded input and output.

The issue is that as an algorithm becomes even slightly complex, the relationship between input and output becomes obscure if represented by hard-coded values. The unit test ends up being a postulate. It may work technically, but hurts test maintenability because it leads to Obscure Tests.

Using Derived Values to test against the result establishes a much clearer relationship between test input and expected output.

The argument that this doesn't test anything is simply not true, because any test case will exercise only a part of a path through the SUT, so no single test case will reproduce the entire algorithm being tested, but the combination of tests will do so.

An additional benefit is that you can use fewer unit tests to cover the desired functionality, and even make them more communicative at the same time. The end result is terser and more maintainable unit tests.

Answer 6

Never use random data for input. If your test reports a failure, how are you ever going to be able to duplicate it? And don't use the same function to generate the expected result. If you have a bug in your method you're likely to put the same bug in your test. Compute the expected results by some other method.

Hard-coded values are perfectly fine, and make sure inputs are picked to represent all of the normal and edge cases. At the very least test the expected inputs as well as inputs in the wrong format or wrong size (eg: null values).

It's really quite simple -- a unit test must test whether the function works or not. That means you need to give a range of known inputs that have known outputs and test against that. There is no universal right way to do that. However, using the same algorithm for the method and the verification proves nothing but that you're adept at copy/paste.

Answer 7

Yes. It bothers me too.. although I'd say that it is more prevalent with non-trivial computations. In order to avoid updating the test when the code changes, some programmers write a IsX=X test, which always succeeds irrespective of the SUT

• About duplicating functionality

You don't have to. Your test can state what the expected output is, not how you derived it. Although in some non-trivial cases, it may make your test more readable as to how you derived the expected value - test as a spec. You shouldn't refactor away this duplication

def doubler(x); x * 2; end

def test_doubler()
  input, expected = 10, doubler(10)

  assert_equal expected, doubler(10)
end

Now if I change doubler(x) to be a tripler, the above test won't fail. def doubler(x); x * 3; end

However this one would:

def test_doubler()
   assert_equal(20, doubler(10))
end

• randomness in unit tests - Don't.

Instead of random datasets, choose static representative data points for testing and use a xUnit RowTest/TestCase to run the test with diff data inputs. If n input-sets are identical for the unit, choose 1. The test in the OP could be used as a exploratory test/ or to determine additional representative input-sets. Unit tests need to be repeatable (See q#61400) - Using random values defeats this objective.