Is Pex (Test generation) really usefull tool?

Question

Yes, it is possible to generate tests on boundary values for functions like "Sum" or "Divide". Pex is a good tool here.

But more often we create tests on business behaviour. Let's consider example from classic Beck's tdd book:

[Test]
public void ShouldRoundOnCreation()
{
Money money = new Money(20.678);
Assert.AreEqual(20.68,money.Amount);
Assert.AreEqual(2068,money.Cents);
}

Can this test be generated? No :) 95 % of tests in my projects check business logic, and can not be generated.

Pex (Especially in pair with Moles) can give 100% code coverage, but a high code coverage rate of a test suite does never indicate, that code is well tested - It only gives false confidence that everything is tested. And this is very dangerous.

So, the question is - Is Pex really usefull tool?

Answer 1

There's a few things useful in Pex.

1. Code coverage. Let's put Pex aside for a second. In general, 100% code coverage doesn't necessarily mean you have good code coverage. It just means ever path is executed, but program has data-flow, and testing that code different, additional inputs, gives you best "test coverage", if not code coverage. Here's, I'm just reiterating your point. 100% code coverage is not necessarily good, but you can say for sure that 25% code coverage is bad, so that's how code coverage is useful. When you have low code coverage, you know for sure you have low test coverage.

   When you use Pex to get 100% code coverage, it's not really helping you get better test coverage per se, but what it does do is give ever piece of the production code some test, which can be used for debugger. In fact, a Pex presentation as a conference shows the use of Pex for this very purpose. The programmer said, "Gee, look at this method in NHibernate. I'd like to step through it in a debugger to see what it does, but how do I even invoke that method through the normal "business-entry point" into the library? Without knowing anything about the library, you can't. So he ran Pex, and was able to step through the code with all sorts of parameters. Interesting, Yes. Useful, maybe so, maybe no.

2. In addition to automatically created tests, Pex is also useful for parameterizing your tests. It's much better to create unit tests that are data driven. Why write the same code over and over, with different parameters. Write it once, and feed the parameters from a data source.

3. Pex is also useful as a simple stubbing framework. It's probably the easiest way to create mock objects, using the new lambda expression, which is much easier to understand than complex frameworks like RhinoMocks, etc. With Moles, you can also stub not just interface, but concrete non-virtual methods in classes.

4. I would also be careful with the "testing at the business logic" layer, too much. You could easy get to doing Behavioral Driven Development, which is not for unit testing. There, you are testing against a spec. If that's all you did, how would you test, say, custom data structures, etc., that have no business value, but are internal libraries used throughout the application.

Answer 2

I think you are mistaken in the way Pex should be used: we highly recommend users to write assertions in their parameterized unit tests.

If you write assertions, Pex will try systematically to invalidate them - that's where the power of Pex comes in: the more you write assertions, the more it tries to find bugs for you.

If you use Pex to get a high code coverage test suite without writing assertions, you only get what you asked for: code coverage and guaranteed runtime exceptions. Pex 'only' tries to cover branches (1 assertion = 1 branch), if there are no branches to cover (no assertion), it won't generate interresting test cases.

In general, writing assertions in parameterized unit tests are harder to write because... well they are more general. Let's start with the rounding behavior: there is certainly a bound on which you allow rounding to occur, this should translate naturally into a parameterized unit test:

[PexMethod]
public void RoundInBounds(double value) 
{
    var money = new Money(value);
    // distance between value and amount should be at most 0.01/2
    Assert.AreEqual(value, money.Amount, 0.005);
}

There are also a number of patterns that can be used to write those. For example, your 'Money' class is probably idempotent: if you feed the value of 'Amount' back in a Money instance, you get Amount back. This translate elegantly into a parameterized unit test:

[PexMethod]
public void RoundIsIdempotent(double value) 
{
     var first = new Money(value).Amount;
     var second = new Money(first).Amount;
     Assert.AreEqual(first, second, 0.0001);
}

Note also, that parameterized unit tests definitely belong in the TDD world. Just write the parameterized unit test first, Pex will find the failing bug, fix the bug, Pex finds the next failing bug, etc...

Does this make Pex a useful tools? You be the judge.