Unit testing , approval testing and datafiles

Question

(Leaving aside hair-splitting about if this is integration-testing or unit-testing.)

I would rather first test at the large scale. If my app writes a VRML file that is the same as the reference one then the VRML exporter works, I don't then have to run unit tests on every single statement in the code. I would also like to use this do some level of poor-man gui testing by comparing screenshots.

Is there a unit test framework (for C++ ideally) that integrates this sort of testing - or at least makes it easy to integrate with unit tests?

edit. It seems a better term is approval testing. So are there any other unit test frameworks that incorporate Approval Testing ?

Answer 1

Kitware, for VTK, uses CDash to do most of its testing. Many of its tests are similar in nature to this - they write out an image of the rendered model, and compare it to a reference image.

In addition, they have code in there to specifically handle very subtle differences to the reference image due to different graphics card drivers/manufacturers/etc. The tests can be written in a way to compare the reference image with some tolerance.

Answer 2

Okay, I think you're making an incorrect assumption about the nature of unit test code; your statement that

If my app writes a VRML file that is the same as the reference one then the VRML exporter works, I don't then have to run unit tests on every single statement in the code.

is strictly correct if you're looking to do a validation test on your code, but note that this type of test is strictly different than what a unit test actually is. Unit tests are for testing individual units of code; they do not exist for verification purposes. Depending on your environment, you may not need unit tests at all, but please keep in mind that validation tests (testing the validity of the overall program output) and unit tests (testing that the individual code units work as expected) are completely different things.

(Note that I'm really not trying to be nitpicky about this; also, you can use plenty of unit test frameworks to achieve this result; keep in mind, though, that what you're writing aren't really "Unit Tests", despite running them in a Unit Test framework.)

Answer 3

Have a look at Approval Tests, written by a couple of friends of mine. Not C++, but it's the general idea of what you're after, also known as Golden Master tests. Good stuff, whether it's unit tests or not.