Unit testing: is "good enough" good enough?

Question

I have a unit test that can spontaneously fail 1 in 1,000,000 (guesstimation) times even when there is no fault in the code. Is this an acceptable tolerance or does the TDD manifesto requires iron fisted absoluteness?

Just for those who are interested it goes something like this

stuff = randomCrap.get()
stuff2 = randomCrap.get()
assert(stuff != stuff2)

Answer 1

Well, that really depends on the source of the failure. Do you know why it fails? If so, have you tried to isolate that fault so that it doesn't trip up the unit test?

Personally, I'd say that if it's genuinely 1 in a million and you know why it's happening, then add a comment to that effect and don't worry about it. It's not likely to bother people significantly in a continuous build, after all. Of course, if it's really one in ten, or something like that, that's a very different matter.

I would at least try to remove the source of incorrectness though. For one thing, it suggests your test isn't repeatable. Sometimes that's okay - there are some sources of randomness which are very difficult to extract out - but I would try not to do it. If you've tried and reached a block, then the pragmatic thing to do is document it and move on, IMO.

Answer 2

What is your requirement for how often before this error should happen?

Are you comfortable, and your functional users, with what is happening?

You may want to make certain it is written down that your random generation will repeat, sometimes in duplicate requests.

To me that is the more troubling part, that it can happen as you showed above, and I would think that this is something to look into.

Answer 3

Do you absolutely have to use random data? Are you testing a system that is made to return two distinct random values? Otherwise create a stub. Unit tests should be 100% repeatable, this is hard to do when for example using threads of file system stuff, but that is what stubs are for.

Greets Jan

Answer 4

A unit test cannot fail if there is no fault in the code. There is a fault, be it related to timing, networking, etc... you just haven't figured it out yet. Once you do figure it out, you just learned something that you didn't know. +1 to you.

The real problem if it is not fixed is psychological. That method will tend to get blamed whenever there is something strange/random/unexplained that happens in the system. Better to fix the red herring now when you are thinking about it.

And FYI, randomness does not imply unique.

Answer 5

The question isn't "does TDD permit me to allow an occasionally failing test?" but, rather: "do my system's requirements permit occasional failures?" And that's a question only you can answer.

From the TDD perspective, it's clumsy to have tests which fail occasionally - you don't know, when they fail, whether it's because this is one of those rare permissible failures, or whether it's because your code is broken in an unacceptable way. So an occasionally failing test is significantly less useful to you than one which always passes.

If your requirement is to have a different behavior one time out of a million, then you should test to that requirement. Test the general case, not with a random number, but with a meaningful subset of valid inputs. Test the special case with the value that should bring about the special behavior.

Answer 6

Your test asserts that stuff1 is never equal to stuff2, but seeing the test fail occasionally means that this is not true.

You might be able to assert that stuff1 is occasionally equal to stuff2 by taking a million samples and asserting that the frequency of being equal is less than 10, but still this will fail occasionally - but perhaps much less often which might be acceptable.

You might be better off with:

stuff = 4
stuff2 = 5
assert(stuff != stuff2)

You can be pretty certain that the above code will perform the same as your original code once in a million million times - but you are certain that this code will pass every time!