I just started a new Haskell project and wanted to set up a good testing workflow from the beginning. It seems like Haskell has a lot of excellent and unique testing tools and many different ways to integrate them.
I have looked into:
Which all seem to work very well in their domains, but I'm looking for a comprehensive approach to testing and was wondering what has worked well for other people.
The approach is advocate in RWH ch 11 and in XMonad is approximately:
Once your major invariants are established via QuickCheck, you can start refactoring, moving those tests into type invariants.
Practices to support your efforts:
The test-framework package is really awesome: http://hackage.haskell.org/package/test-framework-0.3.1. You can easily integrate HUnit and QuickCheck tests, and get executables that run specified suites only, based on command-line flags, with multiple output targets.
Testing and profiling are different beasts though. For profiling I'd set up a separate executable that stresses just the section you want to profile, and just looking carefully at the results of profiling builds and runs (with -prof-auto-all for compilation and +RTS -p for a runtime flag).
Getting unit testing, code coverage, and benchmarks right is mostly about picking the right tools.
I'll use as a running example a package that I just started enabling with unit testing, code coverage, and benchmarks:
http://github.com/ekmett/speculation
You can integrate your tests and benchmarks directly into your cabal file by adding sections for them, and masking them behind flags so that they don't make it so that every user of your library has to have access to (and want to use for themselves) the exact version of the testing tools you've chosen.
http://github.com/ekmett/speculation/blob/master/speculation.cabal
Then, you can tell cabal about how to run your test suite. As cabal test doesn't yet exist -- we have a student working on it for this year's summer of code! -- the best mechanism we have is to use cabal's user hook mechanism. This means switching to a 'Custom' build with cabal and setting up a testHook. An example of a testHook that runs a test program written with test-framework, and then applies hpc to profile it can be found here:
http://github.com/ekmett/speculation/blob/master/Setup.lhs
And then you can use test-framework to bundle up QuickCheck and HUnit tests into one program:
http://github.com/ekmett/speculation/blob/master/Test.hs
The cabal file there is careful to turn on -fhpc to enable code coverage testing, and then the testHook in Setup.lhs manually runs hpc and writes its output into your dist dir.
For benchmarking, the story is a little more manual, there is no 'cabal benchmark' option. You could wire your benchmarks into your test hook, but I like to run them by hand, since Criterion has so many graphical reporting options. You can add your benchmarks to the cabal file as shown above, give them separate compilation flags, hide them behind a cabal flag, and then use Criterion to do all the heavy lifting:
http://github.com/ekmett/speculation/blob/master/Benchmark.hs
You can then run your benchmarks from the command line and get pop-up KDE windows with benchmark results, etc.
Since in practice you're living in cabal anyways while developing Haskell code, it makes a lot of set to integrate your toolchain with it.
For testing, I rely on HUnit and QuickCheck properties and use the Haskell Test Framework to collect all unit tests and all QuickCheck properties automatically.
Disclaimer: I'm the main developer of the Haskell Test Framework.