Have you used Quickcheck in a real project

Question

Quickcheck and its variants (even there is one in java), seems to be interesting. However, apart from academic interest, is it really useful in a real application testing (Eg. a GUI application or Client/Server or even take StackOverflow itself)? Any experiences you had with similar test generators is appreciated.

Answer 1

Yes, well. Actually no, but I've studied under the man who originally developed QuickCheck and he's a really interesting guy.

Back in 2004, we were forced to use QuickCheck to test our Haskell programs and it was combination of good and bad. Mostly bad because Haskell was a bit daunting itself but never the less wonderful when you got it working.

John has since then perfected that which he wrote years back and actually helped Ericssion test their complex telecom hardware, and he found bugs in 20 millions or so lines of code reducing that to a mere three steps through his approach. He's a great speaker so it's always a joy listening to him present what he does so well, but all in all, what he did with QuickCheck was new to me. So I asked him, what his interest was in bringing this to the market. He was open to the idea, but at the time his business (based around QuickCheck) was relatively new and so there were other areas he would focus on. This is now 2007. My point is, you could learn from QuickCheck even if you wont end up using it.

But what is QuickCheck? It's a a combinatorial testing framework and an interesting way to test programs. The people over at Microsoft Research has built Pex which is sort of similar. Pex generates tests automatically by examining your IL. However, John would write a generator for possible input and test properties of a function. A property is something which can easily be tested and it's a lot more formal. e.g. reversing a list? Well, reversing a list, is the same thing as splitting a list in two halves, reversing them each individually and then concatenating the two reversed halves in reverse order.

1,2,3,4 // original
1,2 3,4 // split into A and B
2,1 4,3 // reverse A and B
4,3,2,1 // concat B and A

This is a great property to test with QuickCheck called the specification and the result is quite astonishing.

Pex is nice, but not as cool as QuickCheck, Pex simplifies things, QuickCheck does to but it takes a lot of effort to write a good specification.

The power of QuickCheck is that when it runs into a failure it will reduce the input which caused your test to fail, to the smallest possible form. Leaving you with a detailed description of what progression of state caused your test to fail. In comparison to other testing frameworks which will just try to break your code in a brute force manner.

This is made possible due to how you write your testing specification. QuickCheck relies on pseudo-randomness to invent input and it's because of this, its capable of backtracking and find really small input which does not pass your test.

It's a lot more work to write QuickCheck properties but the end result is better testing. As John himself said, 70% of bugs are caught by unit testing, but it's that other 30% which causes your program to crash. QuickCheck is testing those last 30%.

Answer 2

I've used it quite a lot, mostly in straighforward manner, chiefly for testing protocol and parser implementations.

However, here is less trivial bit from my personal experience: http://www.haskell.org/haskellwiki/QuickCheck_as_a_test_set_generator

Answer 3

I use QuickCheck for a lot of personal stuff. Within the last six months:

• Ran QuickCheck to test color transformations and discrete cosine transforms in an image compressor.

• Ran QuickCheck to test a symbolic-differentiation module I whipped up for some numerical optimization.

• Ran QuickCheck to test a ternary-search tree in the style of Bentley and Sedgewick.

QuickCheck rarely meets all my unit-testing needs, but it's a great way to get started---and the QuickCheck laws make good documentation.

Answer 4

ScalaCheck (a QuickCheck for Scala) is used to test Functional Java, a library that, among other things, implements a QuickCheck for Java.

Answer 5

I've done a real Haskell problem which involved a discrete event simulation. So I wrote a DES library based on the continuation monad, along with the equivalents to MVars and Channels. I needed to check that this worked properly, so I wrote a bunch of QuickCheck properties to demonstrate that, for instance, two streams of concurrent data written to a Channel would be correctly merged without dropping anything.

I've also used QuickCheck to document and verify the properties in my Ranged Sets and Decimal libraries.

In my experience QuickCheck is sometimes great. If you can summarise an important property in a concise way, although the algorithm that delivers that property is hairy, then QuickCheck is a huge win. On the other hand I often find that the algorithm is equivalent to the property I want to verify. In that case I look for simpler properties. For instance, suppose function "foo" is supposed to be non-strictly monotonic. Then you can write

prop_fooMonotonic x y = (x > y) ==> (foo x >= foo y)

Answer 6

I have only used Haskell in a production environment for the development of little helper tools. Mainly because I'm the only developer I know that reads Haskell. I used QuickCheck extensively though, and got really annoyed that something similar is not available in C#. So I decided to try and write it myself. I looked at Pex too, but found the program exploration techniques that are used to find minimal input less interesting than the way QuickCheck does that.