Is my code really not unit-testable?

Question

A lot of code in a current project is directly related to displaying things using a 3rd-party 3D rendering engine. As such, it's easy to say "this is a special case, you can't unit test it". But I wonder if this is a valid excuse... it's easy to think "I am special" but rarely actually the case.

Are there types of code which are genuinely not suited for unit-testing? By suitable, I mean "without it taking longer to figure out how to write the test than is worth the effort"... dealing with a ton of 3D math/rendering it could take a lot of work to prove the output of a function is correct compared with just looking at the rendered graphics.

Answer 1

If you can grab the rendered image, you can unit test it.

Simply render some images with the current codebase, see if they "look right" (examining them down to the pixel if you have to), and store them for comparison. Your unit tests could then compare to those stored images and see if the result is the same.

Whether or not this is worth the trouble, that's for you to decide.

Answer 2

Code that directly relates to displaying information, generating images and even general UI stuff, is sometimes hard to unit-test.

However that mostly applies only to the very top level of that code. Usually 1-2 method calls below the "surface" is code that's easily unit tested.

For example, it may be nontrivial to test that the some information is correctly animated into the dialog box when a validation fails. However, it's very easy to check if the validation would fail for any given input.

Make sure to structure your code in a way that the "untestable" surface area is well-separated from the test and write extensive tests for the non-surface code.

Answer 3

Break down the rendering into steps and test by comparing the frame buffer for each step to a known good images.

No matter what you have, it can be broken down to numbers which can be compared. The real trick is when you havbe some random number generator in the algorithm, or some other nondeterministic part.

With things like floating point, you might need to subtract the generated data from the expected data and check that the difference is less than some error threshold.

Answer 4

If you cannot break your code into units, it is very hard to unit test. My guess would be that if you have 3D atomic functions (say translate, rotate, and project a point) they should be easily testable - create a set of test points and test whether the transformation takes a point to where it should. If you can only reach the 3D code through a limited API, then it would be hard to test. Please see Misko Hevery's Testability posts and his testability guide.

Answer 5

The point of unit-testing your rendering code is not to demonstrate that the third-party-code does the right thing (that is for integration and regression testing). The point is to demonstrate that your code gives the right instructions to the third-party code. In other words, you only have to control the input of your code layer and verify the output (which would become the input of the renderer).

Of course, you can create a mock version of the renderer which does cheap ASCII graphics or something, and then verify the pseudo-graphics if you want and this makes the test clearer if you want, but it is not strictly necessary for a unit test of your code.

Answer 6

I think this is a good question. I wrestle with this all the time, and it seems like there are certain types of code that fit into the unit testing paradigm and other types that do not.

What I consider clearly unit-testable is code that obviously has room for being wrong. Examples:

• Code to compute hairy math or linear algebra functions. I always write an auxiliary function to check the answers, and run it once in a while.

• Hairy data structure code, with cross-references, guids, back-pointers, and methods for incrementally keeping it consistent. These are really easy to break, so unit tests are good for seeing if they are broken.

On the other hand, in code with low redundancy, if the code compiles it may not be clear what being wrong even means. For example, I do pretty complicated UIs using dynamic dialogs, and it's not clear what to test. All the kinds of things like event handling, layout, and showing / hiding / updating of controls that might make this code error-prone are simply dealt with in a well-verified layer underneath.

The kind of testing I find myself needing more than unit-testing is coverage testing. Have I tried all the possible features and combinations of features? Since this is a very large space and it is prohibitive to write automated tests to cover it, I often find myself doing monte-carlo testing instead, where feature selections are chosen at random and submitted to the system. Then the result is examined in an automatic and / or manual way.

Answer 7

Well you can't unit test certain kinds of exception code but other than that ...

I've got true unit tests for some code that looks impossible to even attach a test harness to and code that looks like it should be unit testable but isn't.

One of the ways you know your code is not unit testable is when it depends on the physical characteristics of the device it runs on. Another kind of not unit-testable code is direct UI code (and I find a lot of breaks in direct UI code).

I've also got a huge chunk of non unit-testable code that has appropriate integration tests.