Unit Testing with functions that return random results

Question

I don't think that this is specific to a language or framework, but I am using xUnit.net and C#.

I have a function that returns a random date in a certain range. I pass in a date, and the returning date is always in range of 1 to 40 years before the given date.

Now I just wonder if there is a good way to unit test this. The best approach seems to be to create a loop and let the function run i.e. 100 times and assert that every of these 100 results are in the desired range, which is my current approach.

I also realize that unless I am able to control my Random generator, there will not be a perfect solution (after all, the result IS random), but I wonder what approaches you take when you have to test functionality that returns a random result in a certain range?

Answer 1

Normaly I use exactly your suggested approach: Control the Random generator. Initialize it for test with a default seed (or replace him by a proxy returning numbers which fit my testcases), so I have deterministic/testable behaviour.

Answer 2

You don't need to control the system to make the results deterministic. You're on the right approach: decide what is important about the output of the function and test for that. In this case, it is important that the result be in a range of 40 days, and you are testing for that. It's also important that it not always return the same result, so test for that too. If you want to be fancier, you can test that the results pass some kind of randomness test..

Answer 3

In addition to testing that the function returns a date in the desired range, you want to ensure that the result is well-distributed. The test you describe would pass a function that simply returned the date you sent in!

So in addition to calling the function multiple times and testing that the result stays in the desired range, I would also try to assess the distribution, perhaps by putting the results in buckets and checking that the buckets have roughly equal numbers of results after you are done. You may need more than 100 calls to get stable results, but this doesn't sound like an expensive (run-time wise) function, so you can easily run it for a few K iterations.

I've had a problem before with non-uniform "random" functions.. they can be a real pain, it's worth testing for early.

Answer 4

If you want to check the quality of the random numbers (in terms of independance) there are several ways to do this. One good way is the Chi square test.

Answer 5

Mock or fake out the random number generator

Do something like this... I didn't compile it so there might be a few syntax errors.

public interface IRandomGenerator
{
    double Generate(double max);
}

public class SomethingThatUsesRandom
{
    private readonly IRandomGenerator _generator;

    private class DefaultRandom : IRandomGenerator
    {
        public double Generate(double max)
        {
            return (new Random()).Next(max);
        }
    }

    public SomethingThatUsesRandom(IRandomGenerator generator)
    {
        _generator = generator;
    }

    public SomethingThatUsesRandom() : this(new DefaultRandom())
    {}

    public double MethodThatUsesRandom()
    {
        return _generator.Generate(40.0);
    }
}

In your test, just fake or mock out the IRandomGenerator to return something canned.

Answer 6

I think there are three different aspects of this problem that you test.

The first one: is my algorithm the right one? That is, given a properly-functioning random-number generator, will it produce dates that are randomly distributed across the range?

The second one: does the algorithm handle edge cases properly? That is, when the random number generator produces the highest or lowest allowable values, does anything break?

The third one: is my implementation of the algorithm working? That is, given a known list of pseudo-random inputs, is it producing the expected list of pseudo-random dates?

The first two things aren't something I'd build into the unit-testing suite. They're something I'd prove out while designing the system. I'd probably do this by writing a test harness that generated a zillion dates and performed a chi-square test, as daniel.rikowski suggested. I'd also make sure this test harness didn't terminate until it handled both of the edge cases (assuming that my range of random numbers is small enough that I can get away with this). And I'd document this, so that anyone coming along and trying to improve the algorithm would know that that's a breaking change.

The last one is something I'd make a unit test for. I need to know that nothing has crept into the code that breaks its implementation of this algorithm. The first sign I'll get when that happens is that the test will fail. Then I'll go back to the code and find out that someone else thought that they were fixing something and broke it instead. If someone did fix the algorithm, it'd be on them to fix this test too.

Answer 7

Depending on how your function creates the random date, you may also want to check for illegal dates: impossible leap years, or the 31st day of a 30-day month.

Answer 8

Methods that do not exhibit a deterministic behavior cannot be properly unit-tested,as the results will differ from one execution to another. One way to get around this is to seed the random number generator with a fixed value for the unit test. You can also extract the randomness of the date generation class (and thus applying the Single Responsibility Principle), and inject known values for the unit-tests.

Answer 9

Sure, using a fixed seed random number generator will work just fine, but even then you're simply trying to test for that which you cannot predict. Which is ok. It's equivalent to having a bunch of fixed tests. However, remember--test what is important, but don't try to test everything. I believe random tests are a way to try to test everything, and it's not efficient (or fast). You could potentially have to run a great many randomized tests before hitting a bug.

What I'm trying to get at here is that you should simply write a test for each bug you find in your system. You test out edge cases to make sure your function is running even in the extreme conditions, but really that's the best you can do without either spending too much time or making the unit tests slow to run, or simply wasting processor cycles.

Answer 10

I would recommend overriding the random function. I am unit testing in PHP so I write this code:

// If we are unit testing, then...
if (defined('UNIT_TESTING') && UNIT_TESTING)
{
   // ...make our my_rand() function deterministic to aid testing.
   function my_rand($min, $max)
   {
      return $GLOBALS['random_table'][$min][$max];
   }
}
else
{
   // ...else make our my_rand() function truly random.
   function my_rand($min = 0, $max = PHP_INT_MAX)
   {
      if ($max === PHP_INT_MAX)
      {
         $max = getrandmax();
      }
      return rand($min, $max);
   }
}

I then set the random_table as I require it per test.

Testing the true randomness of a random function is a separate test altogether. I would avoid testing the randomness in unit tests and would instead do separate tests and google the true randomness of the random function in the programming language you are using. Non-deterministic tests (if any at all) should be left out of unit tests. Maybe have a separate suite for those tests, that requires human input or much longer running times to minimise the chances of a fail that is really a pass.