Using randomness and/or iterations in unit tests?

Question

In unit tests, I have become used to test methods applying some regular values, some values offending the method contract, and all border-cases I can come up with.

But is it very bad-practice to

• test on random values, this is a value within a range you think should never give any trouble, so that each time the test runs, another value is passed in? As a kind of extensive testing of regular values?
• test on whole ranges, using iteration ?

I have a feeling both of this approaches aren't any good. With range-testing I can imagine that it's just not practical to do that, since the time it is taking, but with randomness?

UPDATE :

I'm not using this technique myself, was just wondering about it. Randomness can be a good tool, I know now, if you can make it reproduceable when you need to.
The most interesting reply was the 'fuzzing' tip from Lieven :

http://en.wikipedia.org/wiki/Fuzz_testing

tx

Answer 1

What are you testing? The random number generator? Or your code?

If your code, what if there is a bug in the code that produces random numbers?

What if you need to reproduce a problem, do you keep restarting the test hoping that it will eventually use the same sequence as you had when you discovered the problem?

If you decide to use a random number generator to produce data, at least seed it with a known constant value, so it's easy to reproduce.

In other words, your "random numbers" are just a "sequence of numbers I really don't care all that much about".

Answer 2

Unit tests need to be fast. if they aren't people won't run them regularly. At times I did code for checking the whole range but @Ignore'd commented it out in the end because it made the tests too slow. If I were to use random values, I would go for a PRNG with fixed seeds so that every run actually checks the same numbers.

Answer 3

So long as it will tell you in some way what random value it failed on I don't suppose it's that bad. However, you're almost relying on luck to find a problem in your application then.

Testing the whole range will ensure you have every avenue covered but it seems like overkill when you have the edges covered and, I assume, a few middle-ground accepted values.

Answer 4

1. Random Input - The tests would not be repeatable (produce consistent results every time they are run and hence are not considered good unit tests. Tests should not change their mind.
2. Range tests / RowTests - are good as long as they dont slow down the test suite run.. each test should run as fast as possible. (A done-in-30sec test suite gets run more often than a 10 min one) - preferably 100ms or less. That said Each input (test data) should be 'representative' input. If all input values are the same, testing each one isn't adding any value and is just routine number crunching. You just need one representative from that set of values. You also need representatives for boundary conditions and 'special' values.

For more on guidelines or thumbrules - see 'What makes a Good Unit Test?'

That said... the techniques you mentioned could be great to find representative inputs.. So use them to find scenarioX where code fails or succeeds incorrectly - then write up a repeatable,quick,tests-one-thing-only unit test for that scenarioX and add it to your test suite. If you find that these tools continue to help you find more good test-cases.. persist with them.

Response to OP's clarification:

• If you use the same seed value (test input) for your random no generator on each test run, your test is not random - values can be predetermined. However a unit test ideally shouldn't need any input/output - that is why xUnit test cases have the void TC() signature.
• If you use different seed values on each run, now your tests are random and not repeatable. Of course you can hunt down the special seed value in your log files to know what failed (and reproduce the error) but I like my tests to instantly let me know what failed - e.g. a Red TestConversionForEnums() lets me know that the Enum Conversion code is broken without any inspection.

Repeatable - implies that each time the test is run on the SUT, it produces the same result (pass/fail).. not 'Can I reproduce test failure again?' (Repeatable != Reproducible). To reiterate.. this kind of exploratory testing may be good to identify more test cases but I wouldn't add this to my test suite that I run each time I make a code change during the day. I'd recommend doing exploratory testing manually, find some good (some may use sadistic) Testers that'll go hammer and tongs at your code.. will find you more test cases than a random input generator.

Answer 5

I wouldn't advocate completely random values as it will give you a false sense of security. If you can't go through the entire range (which is often the case) it's far more efficient to select a subset by hand. This way you will also have to think of possible "odd" values, values that causes the code to run differently (and are not near edges).

You could use a random generator to generate the test values, check that they represent a good sample and then use them. This is a good idea especially if choosing by hand would be too time-consuming.

I did use random test values when I wrote a semaphore driver to use for a hw block from two different chips. In this case I couldn't figure out how to choose meaningful values for the timings so I randomized how often the chips would (independently) try to access the block. In retrospect it would still have been better to choose them by hand, because getting the test environment to work in such a way that the two chips didn't align themselves was not as simple as I thought. This was actually a very good example of when random values do not create a random sample.

The problem was caused by the fact that whenever the other chip had reserved the block the other waited and true to a semaphore got access right after the other released it. When I plotted how long the chips had to wait for access the values were in fact far from random. Worst was when I had the same value range for both random values, it got slightly better after I changed them to have different ranges, but it still wasn't very random. I started getting something of a random test only after I randomized both the waiting times between accesses and how long the block was reserved and chose the four sets carefully.

In the end I probably ended up using more time writing the code to use "random" values than I would have used to pick meaningful values by hand in the first place.

Answer 6

What you describe is usually called specification-based testing and has been implemented by frameworks such as QuickCheck (Haskell), scalacheck (Scala) and Quviq QuickCheck (Erlang).

Data-based testing tools (such as DataProvider in TestNG) can achieve similar results.

The underlying principle is to generate input data for the subject under test based upon some sort of specification and is far from "bad practice".

Answer 7

The goal of unit-testing is to get confidence in your code. Therefore, if you feel that using random values could help you find some more bugs, you obviously need more tests to increase your confidence level.

In that situation, you could rely on iteration-based testing to identify those problems. I'd recommend creating new specific tests for the cases discovered with the loop testing, and removing the iteration-based tests then; so that they don't slow down your tests.

Answer 8

I have used randomness for debugging a field problem with a state machine leaking a resource. We code inspected, ran the unit-tests and couldn't reproduce the leak.

We fed random events from the entire possible event space into the state machine unit test environment. We looked at the invariants after each event and stopped when they were violated.

The random events eventually exposed a sequence of events that produced a leak. The state machine leaked a resource when a 2nd error occurred while recovering from a first error.

We were then able to reproduce the leak in the field.

So randomness found a problem that was difficult to find otherwise. A little brute force but the computer didn't mind working the weekend.

Answer 9

I have been using randomness in my testcases. It found me some errors in the SUT and it gave me some errors in my testcase.

Note that the testcase get more complex by using randomnes.

• You'll need a method to run your testcase with the random value(s) it failed on
• You'll need to log the random values used for every test.
• ...

All in all, I'm throthling back on using randomness but not dismissing it enterly. As with every technique, it has its values.

For a better explanation of what you are after, look up the term fuzzing

Answer 10

See David Saff's work on Theory-Based Testing.

Generally I'd avoid randomness in unit tests, but the theory stuff is intriguing.

Answer 11

The 'key' point here is unit test. A slew of random values in the expected ranged as well as edges for the good case and ouside range/boundary for bad case is valuable in a regression test, provided the seed is constant.

a unit test may use random values in the expected range, if it is possible to always save the inputs/outputs (if any) before and after.