Code coverage targets for APIs

Question

What number would you give someone who wants a concrete target number for API code coverage?

UPDATE: To clarify, statement/line code coverage. I realize concrete numbers don't make much sense, but this is for the situation where you tell people that concrete numbers don't make much sense and they still insist on getting a number from you no matter what. I specifically wrote API/SDK because some people might find lower code coverages more acceptable for application/GUI level software, as opposed to libraries, where more interfaces are exposed.

Answer 1

Could you be a little more specific? What exactly do you mean? Are there any particular situations you refer to?

Answer 2

Concrete numbers don't make much sense, actually.

If there are 14 equivalent cases, is your testing more given to error if you don't test all 14? What if you have a bounds check that should be impossible to fail (I like to throw descriptive exceptions in those cases, which are then emailed to our dev staff)?

Better to just make sure all logical paths are covered.

For a far more detailed answer, see this (very similar) question.

Answer 3

I'd give them the advice that they need to examine the parts of their API to determine which are practically inconsequential and need no more than 20% and which parts are super critical and need > 90%.

Answer 4

Take a look at C.R.A.P. -- it combines coverage with complexity analysis. There is a Crap4J implementation if you are Java. We have been putting together a Crap4Net that we blogged about here:

http://www.atalasoft.com/cs/blogs/insertqualityhere/archive/2008/03/28/crap4j-port-to-net.aspx

The idea is that you get good numbers if you have small simple methods or if the more complex ones have good coverage.

Answer 5

Just forget about code coverage. It's just a number and must not be the focus when testing code. Scenarios must be the focus and then, high Quality API. I know that this could sound rhetorical bullshit, but you must change your mindset from code coverage to scenarios: are you testing a lot of scenarios which your API is intent to handle?

Code coverage will be useful to detect that you are missing some scenarios and if you write a lot of good scenarios you will have a near 100% coverage, but again, it is just a number and must not be your focus.

Kind Regards

Answer 6

You can get some code coverage tools to provide you with "function coverage", e.g, whether or not a function was executed. I'd insist that every function in the API was covered.

Coverage on lines inside the API's implementation is a different matter. Good practices suggest shooting for 70-80% coverage based on lines or statements and total size.

Answer 7

If you're on PHP, 80% seems to be a good recommendation: http://jordionsoftware.blogspot.com/2009/09/code-coverage-targets.html