What's the best way to test this?

Question

I'm going through the EdgeCase Ruby Koans. In about_dice_project.rb, there's a test called "test_dice_values_should_change_between_rolls", which is straightforward:

  def test_dice_values_should_change_between_rolls
    dice = DiceSet.new

    dice.roll(5)
    first_time = dice.values

    dice.roll(5)
    second_time = dice.values

    assert_not_equal first_time, second_time,
      "Two rolls should not be equal"
  end

Except for this comment that appears there:

# THINK ABOUT IT:
#
# If the rolls are random, then it is possible (although not
# likely) that two consecutive rolls are equal.  What would be a
# better way to test this.

Which (obviously) got me thinking: what is the best way to reliably test something random like that (specifically, and generally)?

Answer 1

It seems like there are 2 separate units here. First, a random number generator. Second, a "dice" abstraction that uses the (P)RNG.

If you want to unit test the dice abstraction, then mock out the PRNG calls, and make sure it calls them, and returns an appropriate value for the input you give, etc.

The PRNG is probably part of your library/framework/OS and so I wouldn't bother testing it. Maybe you'll want an integration test to see if it returns reasonable values, but that's a whole 'nother problem.

Answer 2

I'd say the best way to test anything that involves randomness is statistically. Run your dice function in a loop a million times, tabulate the results, and then run some hypothesis tests on the results. A million samples should give you enough statistical power that almost any deviations from correct code will be noticed. You are looking to demonstrate two statistical properties:

1. The probability of each value is what you intended it to be.
2. All rolls are mutually independent events.

You can test whether the frequencies of the dice rolls are approximately correct using Pearson's Chi-square test. If you're using a good random nunber generator, such as the Mersenne Twister (which is the default in the standard lib for most modern languages, though not for C and C++), and you're not using any saved state from previous rolls other than the Mersenne Twister generator itself, then your rolls are for all practical purposes independent of one another.

As another example of statistical testing of random functions, when I ported the NumPy random number generators to the D programming language, my test for whether the port was correct was to use the Kolmogorov-Smirnov test to see whether the numbers generated matched the probability distributions they were supposed to match.

Answer 3

There is no way to write a state-based test for randomness. They are contradictory, since state-based tests proceed by giving known inputs and checking output. If your input (random seed) is unknown, there is no way to test.

Luckily, you don't really want to test the implementation of rand for Ruby, so you can just stub it out with an expectation using mocha.

def test_roll
  Kernel.expects(:rand).with(5).returns(1)
  Diceset.new.roll(5)
end

Answer 4

It seems a bit silly, to me. Are you supposed to be testing that the (psuedo) random number generator is generating random numbers? That's futile and pointless. If anything, you could test that dice.roll calls to your PRNG.

Answer 5

My solution was to allow a block to be passed to the roll function.

class DiceSet
  def roll(n)
    @values = (1..n).map { block_given? ? yield : rand(6) + 1 }
  end
end

I can then pass my own RNG into the tests like this.

dice = DiceSet.net
dice.roll(5) { 1 }
first_result = dice.values
dice.roll(5) { 2 }
second_result = dice.values
assert_not_equal first_result, second_result

I don't know if that's really better, but it does abstract out the calls to the RNG. And it doesn't change the standard functionality.