Unit-testing coverage policy, test quality concerns

Question

Hi,

our company is trying to increase software quality by enforcing minimum function coverage on automated unit-tests. This is already a good starting point to get at least some tests written and to get them automated (although it would be better to go for branch or decision coverage).

My main concern is the outcome of the tests which are going to be written after this policy is taken into use. I have seen too often such rules causing huge amount of null testing (i.e. nothing is asserted) or some maintenance nightmare kind of integration tests. I found following SO questions close to the subject but these concentrate more on coverage percentages:

http://stackoverflow.com/questions/695811/pitfalls-of-code-coverage

http://stackoverflow.com/questions/90002/what-is-a-reasonable-code-coverage-for-unit-tests-and-why

Instead, I would like to get help or insight, how we could avoid awful quality of tests. So here's a couple of most worst unit-test no-nos and what I have already thought for avoiding them:

1) Null testing

• Code review for test code also
• As we use testing framework, assertation are done by using FW macros. Maybe we could write some tool which would calculate ratio for assertation per method call of class under testing. No idea yet, what would be good enough ratio.

2) Integration tests

• Again review
• Perhaps some code analysis tool to check dependencies of test code to production code. Test class should have very little amount of dependencies to other classes (except to the one under testing) of the tested system.

There's plenty of teams and I am not totally convinced that team-internal reviews would be enough in all cases. Therefore I would be more interested in ways of automating quality ensurance.

TIA, lutku

Answer 1

The best way to get good tests is to practice test drivin development, that means test FIRST. It breaks down to a few steps:

1. Write just enough of a test to fail
2. Write just enough production code to get the test to pass
3. 3 repeat, and refractor as you go

Of course this is a huge simplification and TDD is a big big subject, but I have not seen good fast easy to maintain tests without it. The other key is that the developers need to WANT to do it, they need to understand how it makes their lives easier, how it frees them to changing code later. If the developers are not psyched to start testing, or if they don't know how to write tests (managing dependencies following SOLID and FIRST principles) then it may not matter what rules you impose.

Answer 2

There's a metric called CRAP which combines complexity analysis with coverage

http://blog.businessofsoftware.org/2007/09/alberto-savoia-.html

Basically, each function has a score of how bad it is where functions get worse if they are complex or don't have coverage. This means that you can't really lower your score by testing get/set methods -- you have to look at the complex functions. You can either break them down (refactor) or test them to get coverage.

The video explains why it's a hard metric to game.

It won't help with Null tests, but it will help focus the testing on where you get more out of it.

Answer 3

yes, this is a very hard problem (see also blog-post avoid false-positives vs. false-negatives.

Above from the techniques discussed above, there is mutation-testing, where you can check test-suite quality. On some references of the wiki page you can see some hints how to help yourself with custom extensions to the existing frameworks. I once tried out Jester, it was nice but also slow, unfortunately the project currently doesn't seem to be that active...

Answer 4

I've been through this, helping take a company's automated testing from scattergun / patchy through to being largely useful with much higher quality.

I find that trying to apply metrics as the primary driver of quality misses the point. First and foremost, it's a people problem. You can't make someone believe in something without reason, just as you can't magically make them good at it without support.

The short and difficult answer is that, to get the test quality up, you need to treat test code as a first class citizen. People won't make a good job of automated testing unless they can be sold on it and are given the support to improve their skills. Many people skirt around the issue, but the fact is that automated testing is hard to do well, and a lot of developers will not 'get it' or accept that it is a skill to be learned; even worse, many will silently struggle and refuse to ask for help.

Failing to prove its benefits results in lacklustre testing which feeds back to the developers in making them think testing is useless and finds no bugs. If a developer treats testing as a chore and phones it in, they are already in the mindset that it is useless -- it becomes a self-fulfilling prophecy and a total drudge. I know from experience that there is pretty much nothing worse than writing your code then writing all of your tests to hit a magical coverage target. By that time, not only is the code untestable, but it's akin to doing all of your school homework on a Sunday night -- it's no fun.

You need to build awareness of why unit testing can help, what a good/correct/understandable/maintainable/etc. unit test looks like. You can do this through education, presentations, pair programming, code reviews and so forth. If you just set a hard limit and tell people you expect them to meet it, they will probably resent it and then game the system. Note: This is not an easy thing to do! It takes a lot of time to get suspicious developers to see the value in it and start writing non-crazy tests. There is no single 'aha' moment you can bank on. Studies have been done that prove automated testing can be a valuable development tool, but a lot of developers are dogmatic. Some will just never ever come around to the idea.

I find pair programming works pretty well. Find an isolated component that can be easily tested, or write some tests with them. Go through the process of writing tests, making them pass, refactoring the tests and the production code to remove problems and make them more readable. Over time, build up their skills showing them the most common techniques from the testing toolbox. As time goes on, try various techniques such as using good naming practices, named consts, factory methods, test data builders, BDD style 'fixture as context'. Show them how to prove a bug exists via writing a failing test before fixing the bug. Emphasise the most important tenets of creating good tests!

The eventual goal should be that all code teams agree on some rules of thumb such as "To get sign off, all stories must be adequately tested and pass a code review." If automated testing is not valuable / feasible for a given piece of work (e.g. a prototype) that's 100% fine, but it should be the exception, not the rule.

Having respected code leads who will work with their teams to make this happen is of paramount importance. If you cannot get buy in from all of the leads, then you have a major problem.

You can augment your approach using code metrics tools like NDepend (or the variant for your language of choice) which offers features like listing the most complex / used code that does not have good test coverage, or the areas lacking coverage that have changed the most between checkins.

Good luck.