How to test for sometimes fails?

Question

I'm trying to write a unit test for a module that will give me a random list of numbers given certain criteria.

The particular test I'm writing is a reshuffle of the original sequence. I'm testing that

• The sequences are the same length
• The sequences have the same values
• The sequences are not in the same order

The problem with this is that sometimes the sequences are in the same order. What would be the best way to deal with this?

I'm using NUnit (but could use another test framework if it helps).

Answer 1

What would be the best way to deal with this?

There really isn't a way to "deal with it" until the requirements are firmed up. Either it's a requirement that the shuffle ensures the sequences are out of order (in which case the shuffle isn't truly "random") or it's okay that they're in the same order. If it's acceptable for the shuffle to return something in the same order, you can write a test that will check the distribution of results over a large number of runs against an expected distribution, but that's about as specific as you can get for testing "random" results. If it's not okay for the shuffle to return the same order, then you can run the function a number of times to give you an acceptable level of confidence that the shuffle will not return the same order, so the same general principle applies. Use a large number of runs and measure the results against accepted tolerances.

Hope this helps!

Answer 2

How are you generating your random numbers?

I'd start by controlling the seed of your random number generator, and redefining your 3rd test to be "the sequences are not in the same order, when we use a different seed".

Answer 3

• The particular test I'm writing is a reshuffle of the original sequence.
• The sequences are not in the same order

Those requirements don't match. If you are truely randomly reshuffling, then clearly the list can come back unchanged.

There is an option though. When you create your random number generator, you can pass in a "seed". If you always pass in the same seed, you should always get the same answer. Now that you've taken the random out of your random function, you can unit test it easily.