Unit testing - how do I test a function that returns random output?

Question

I have a function which takes in two parameters, and returns one or the other 50% of the time.

The unit test for this should determine that both parameters could be returned. Luckily, I don't need to prove that the probability for each is 50% but I do need to show that both parameters are possible to be returned.

How do I write a test case for this function?

Answer 1

Not sure if this is the best way, but I would use a while loop with two flags that I set based on the values I get from the function. I would also use some sort of limit so that the test doesn't go into an infinite loop if the test fails (i.e., doesn't received one of the two values). So:

int limit = 100;
int i = 0;
while(!firstValueReceived && !secondValueReceived && i < limit) {
   int value = randomFunc();

   if(!firstValueReceived) {
      firstValueReceived = (value == firstValue);
   }

   if(!secondValueReceived) {
      secondValueReceived = (value == secondValue);
   }
   i++;
}

assertTrue(firstValueReceived && secondValueReceived);

I guess you could use the test hanging as a metric to decide if it failed or not, so you could do away with the limit:

while(!firstValueReceived && !secondValueReceived) {
   int value = randomFunc();

   if(!firstValueReceived) {
      firstValueReceived = (value == firstValue);
   }

   if(!secondValueReceived) {
      secondValueReceived = (value == secondValue);
   }
}

assertTrue(firstValueReceived && secondValueReceived);

Answer 2

If the randomness is based on a random number generator that it calls, you can rig up a stub random() function that returns various results to give you your expected outputs.

Answer 3

If the decision is truly random (or pseudo-random), then you can separate out from your function the concern of choosing one outcome from the other. For example, your function could accept as a parameter a delegate or another object whose only job is to "flip a coin" but has no concern over the two parameters or any logic about them. Then your unit test can provide a mock decision-maker object, that returns whatever outcome the test wants to check.

Answer 4

The unit test result isn't a single value for a given response, it's a distribution of values for a range of inputs. Probability means you'll have to run many tests to build this up.

If your object really ought to return one value half the time on average and the other the rest of the time, then the appropriate test would be to run many tests and make sure that the limiting behavior gives you the distribution you want. Build up the appropriate distribution of results and make sure it matches what you expect.

Answer 5

You test it through separation and/or injection:

// ** WARNING: Untested code

Original

void funcToTest(String a, String b) {
  int i = new Random().nextInt(2);
  return ((i == 0) ? a : b);
}

Separating

void funcToTest(String a, String b) {
  return funcToTest(a, b, (new Random().nextInt(2) == 0));
}
void funcToTest(String a, String b, boolean r) {
  return (r ? a : b);
}

Now test

funcToTest(String a, String b, boolean r).

You're trusting that, if you test the above, then

funcToTest(String a, String b)

will work just work.

Injecting

interface BoolRandomizer {
  bool getBool();
}
BoolRandomizer BOOL_RANDOMIZER = new BoolRandomizer() {
  public bool getBool() { return (new Random().nextInt(2) == 0); }
}
BoolRandomizer ALWAYS_FALSE_RANDOMIZER = new BoolRandomizer() {
  public bool getBool() { return false; }
}
BoolRandomizer ALWAYS_TRUE_RANDOMIZER = new BoolRandomizer() {
  public bool getBool() { return true; }
}

class ClassUnderTest {
    private final BoolRandomizer br;
    ClassUnderTest() { this(BOOL_RANDOMIZER); }
    ClassUnderTest(BoolRandomizer br) { this.br = br; }
    void funcToTest(String a, String b) {
      return (br.getBool() ? a : b);
    }
}

Now test

new ClassUnderTest(ALWAYS_FALSE_RANDOMIZER).funcToTest()

and

new ClassUnderTest(ALWAYS_TRUE_RANDOMIZER).funcToTest()

There are of course more complex/slicker ways to do this, but this is the basic idea. I used these techniques for a deck of cards program to test the shuffle function. Another case: a function that uses current time by calling system API gettime() could be hard to test unless you are willing to run the unit test at very specific times of day :-). Separation and injection ease testing.

Answer 6

You can do one of two things:

1. make a stub random() generator that can return consistent numbers when unit testing.
2. run your code N times, where N is high enough for you to get consistent results so long as the code is working.

Option #1 is nearly always preferred. By including your random number generator you're no longer testing an atomic unit of code, and by including a stochastic component into unit testing, you need to do statistics to verify that the code is working correctly. There are some bugs that option #2 can never pick up in a reasonable number of tests, which is worrying.