I'm relatively new to JUnit, and I was writing a few of my first tests today. For some particular method, I wanted to pass random values (all of which are within the correct range). If the method fails for any reason, I want to know which value caused it to fail. So what's the recommended way of doing this?
(Or is it bad to use random values in JUnit tests?)
You can try to use: http://www.openfuture.de/Log4Unit/ for logging, but I would recommend against random values for unit tests as they are supposed to be repeated. If you want to test a lot of values just use a for loop and some modifications to the index value, which is easily repeated.
If you think about it there is really no situation where it would be more beneficial to use random values than "hard coded" ones. If you want a good spread over a value range you can use a function or use random values with a fixed seed (to get the same numbers).
If a test fails you want to be able to fix it and run the test again. That's the problem with random numbers in unit tests.
If your unit test fails for any reason, you will see a red traffic light in your test runner. The test runner will also show you which test method failed and the test runner's log reports more detail (the dumped stack trace, for example). Investigate that error, correct it, and your test should never fail again, unless you break the code.
Thus, I don't see the necessity for logging exceptions. You should fix any error (red traffic light) immediately.
Using random values can be quite dangerous if you cannot guarantee that your generation of these values is free of errors. Checking the boundary conditions might be more useful.
Just report the actual and expected values in the "diagnostic message" parameter of your assertions. That's the common practice for JUnit tests, and the "helper" assert methods tend to do this by default, e.g. assertEquals will say something like "expected 6 and got 7".
Random values are OK, as long as you can ensure that the random range constitutes an equivalence class for the code under test.
If you really want to use random values then simply place the value in use in the textual part of the assert methods. Then if an assert is blown the input value will be present and you can investigate why it had a problem.
This is Fuzz Testing and is a powerful technique but is most useful when you don't have the available source code or when testing a system with complex internal state and multiple interactions.
A more useful type of testing for you may be white box testing where test inputs are deliberately selected to cover the various classes of input you might get. JTest appears to be an automated tool for this in java. MS Research supplies PEX for c#).
Simply using a coverage tool and validating that you cover the relevant paths is normally sufficient if doing it by hand, though the boundary cases supplied by the automated tools are often instructive.
Have you tried using the 'assertThat' methods and Hamcrest Matchers that are part of JUnit 4.4+? Check out the README [1] and search for 'assertThat'.
I've become quite fond of the much more semantic look of the code and the failure messages are much more informative.
You can have repeatable random values by providing a constant seed to the random number generator. In Java create a java.util.Random with a fixed seen and pass to the class as a constructor parameter (dependency injection). Like this:
new ClassUnderTest(new Random(123L));
Depending on what you are testing, you might also separate the generating of the random values from their use. If you have class X which takes random values from range 1 to 10, then you can test it by passing it the edge values 1 and 10, and some value from the middle such as 4. Then you need another test for the producer of those random values. For example give it a java.util.Random with a fixed seed and generate 100 values, and check that all of them are within the allowed range.
I would propose parameterized test cases. so you can use random values (in the Data method) and it's "logged" in your runner, if any method will fail.