Unit testing code coverage - do you have 100% coverage?

Question

Do your unit tests constitute 100% code coverage? Yes or no, and why or why not.

Answer 1

It is seldom practical to get 100% code coverage in a non-trivial system. Most developers who write unit tests shoot for the mid to high 90's.

An automated testing tool like Pex can help increase code coverage. It works by searching for hard-to-find edge cases.

Answer 2

No, because there is a practical trade-off between perfect unit tests and actually finishing a project :)

Answer 3

No because I spent my time adding new features that help the users rather than tricky to write obscure tests that deliver little value. I say unit test the big things, subtle things and things that are fragile.

Answer 4

No for several reasons :

• It is really expensive to reach the 100% coverage, compared to the 90% or 95% for a benefit that is not obvious.
• Even with 100% of coverage, your code is not perfect. Take a look at this method (in fact it depends on which type of coverage you are talking about - branch coverage, line coverage...):

public static String foo(boolean someCondition) {
    String bar = null;
    if (someCondition) {
        bar = "blabla";
    }
    return bar.trim();
}

and the unit test:

assertEquals("blabla", foo(true));

The test will succeed, and your code coverage is 100%. However, if you add another test:

assertEquals("blabla", foo(false));

then you will get a NullPointerException. And as you were at 100% with the first test, you would have not necessarily write the second one!

Generally, I consider that the critical code must be covered at almost 100%, while the other code can be covered at 85-90%

Answer 5

I personally find 100% test coverage to be problematic on multiple levels. First and foremost, you have to make sure you are gaining a tangible, cost-saving benefit from the unit tests you write. In addition, unit tests, like any other code, are CODE. That means it, just like any other code, must be verified for correctness and maintained. That additional time verifying additional code for correctness, and maintaining it and keeping those tests valid in response to changes to business code, adds cost. Achieving 100% test coverage and ensuring you test you're code as thoroughly as possible is a laudable endeavor, but achieving it at any cost...well, is often too costly.

There are many times when covering error and validity checks that are in place to cover fringe or extremely rare, but definitely possible, exceptional cases are an example of code that does not necessarily need to be covered. The amount of time, effort (and ultimately money) that must be invested to achieve coverage of such rare fringe cases is often wasteful in light of other business needs. Properties are often a part of code that, especially with C# 3.0, do not need to be tested as most, if not all, properties behave exactly the same way, and are excessively simple (single-statement return or set.) Investing tremendous amounts of time wrapping unit tests around thousands of properties could quite likely be better invested somewhere else where a greater, more valuable tangible return on that investment can be realized.

Beyond simply achieving 100% test coverage, there are similar problems with trying to set up the "perfect" unit. Mocking frameworks have progressed to an amazing degree these days, and almost anything can be mocked (if you are willing to pay money, TypeMock can actually mock anything and everything, but it does cost a lot.) However, there are often times when dependencies of your code were not written in a mock-able way (this is actually a core problem with the vast bulk of the .NET framework itself.) Investing time to achieve the proper scope of a test is useful, but putting in excessive amounts of time to mock away everything and anything under the face of the sun, adding layers of abstraction and interfaces to make it possible, is again most often a waste of time, effort, and ultimately money.

The ultimate goal with testing shouldn't really be to achieve the ultimate in code coverage. The ultimate goal should be achieving the greatest value per unit time invested in writing unit tests, while covering as much as possible in that time. The best way to achieve this is to take the BDD approach: Specify your concerns, define your context, and verify the expected outcomes occur for any piece of behavior being developed (behavior...not unit.)

Answer 6

I generally write unit tests just as a regression-prevention method. When a bug is reported that I have to fix, I create a unit test to ensure that it doesn't re-surface in the future. I may create a few tests for sections of functionality I have to make sure stay intact (or for complex inter-part interactions), but I usually want for the bug fix to tell me one is necessary.

Answer 7

Yes we do.

It depends on what language and framework you're using as to how easy that is to achieve though.

We're using Ruby on Rails for my current project. Ruby is very "mockable" in that you can stub/mock out large chunks of your code without having to build in overly complicated class composition and construction designs that you would have to do in other languages.

That said, we only have 100% line coverage (basically what rcov gives you). You still have to think about testing all the required branches.

This is only really possible if you include it from the start as part of your continuous integration build, and break the build if coverage drops below 100% - prompting developers to immediately fix it. Of course you could choose some other number as a target, but if you're starting fresh, there isn't much difference for the effort to get from 90% to 100%

We've also got a bunch of other metrics that break the build if they cross a given threshold as well (cyclomatic complexity, duplication for example) these all go together and help reinforce each other.

Again, you really have to have this stuff in place from the start to keep working at a strict level - either that or set some target you can hit, and gradually ratchet it up till you get to a level you're happy with.

Does doing this add value? I was skeptical at first, but I can honestly say that yes it does. Not primarily because you have thoroughly tested code (although that is definitely a benefit), but more in terms of writing simple code that is easy to test and reason about. If you know you have to have 100% test coverage, you stop writing overly complex if/else/while/try/catch monstrosities and Keep It Simple Stupid.

Answer 8

I only have 100% coverage on new pieces of code that have been written with testability in mind. With proper encapsulation, each class and function can have functional unit tests that simultaneously give close to 100% coverage. It's then just a matter of adding some additional tests that cover some edge cases to get you to 100%.

You shouldn't write tests just to get coverage. You should be writing functional tests that test correctness/compliance. By a good functional specification that covers all grounds and a good software design, you can get good coverage for free.

Answer 9

Yes, I have had projects that have had 100% line coverage. See my answer to a similar question.

You can get 100% line coverage, but as others have pointed out here on SO and elsewhere on the internet its maybe only a minimum. When you consider path and branch coverage, there's a lot more work to do.

The other way of looking at it is to try to make your code so simple that its easy to get 100% line coverage.

Answer 10

I usually manage to hit 93..100% with my coverage but I don't aim for 100% anymore. I used to do that and while it's doable, it's not worth the effort beyond a certain point because testing blindly obvious usually isn't needed. Good example of this could be the true evaluation branch of the following code snipped

public void method(boolean someBoolean) {
    if (someBoolean) {
        return;
    } else {
        /* do lots of stuff */ 
    }
}

However what's important to achieve is to as close to 100% coverage on functional parts of the class as possible since those are the dangerous waters of your application, the misty bog of creeping bugs and undefined behaviour and of course the money-making flea circus.

Answer 11

What I do when I get the chance is to insert statements on every branch of the code that can be grepped for and that record if they've been hit, so that I can do some sort of comparison to see which statements have not been hit. This is a bit of a chore, so I'm not always good about it.

I just built a small UI app to use in charity auctions, that uses MySQL as its DB. Since I really, really didn't want it to break in the middle of an auction, I tried something new.

Since it was in VC6 (C++ + MFC) I defined two macros:

#define TCOV ASSERT(FALSE)
#define _COV ASSERT(TRUE)

and then I sprinkled

TCOV;

throughout the code, on every separate path I could find, and in every routine. Then I ran the program under the debugger, and every time it hit a TCOV, it would halt. I would look at the code for any obvious problems, and then edit it to _COV, then continue. The code would recompile on the fly and move on to the next TCOV. In this way, I slowly, laboriously, eliminated enough TCOV statements so it would run "normally".

After a while, I grepped the code for TCOV, and that showed what code I had not tested. Then I went back and ran it again, making sure to test more branches I had not tried earlier. I kept doing this until there were no TCOV statements left in the code.

This took a few hours, but in the process I found and fixed several bugs. There is no way I could have had the discipline to make and follow a test plan that would have been that thorough. Not only did I know I had covered all branches, but it had made me look at every branch while it was running - a very good kind of code review.

So, whether or not you use a coverage tool, this is a good way to root out bugs that would otherwise lurk in the code until a more embarrasing time.

Answer 12

From Ted Neward blog.

By this point in time, most developers have at least heard of, if not considered adoption of, the Masochistic Testing meme. Fellow NFJS'ers Stuart Halloway and Justin Gehtland have founded a consultancy firm, Relevance, that sets a high bar as a corporate cultural standard: 100% test coverage of their code. Neal Ford has reported that ThoughtWorks makes similar statements, though it's my understanding that clients sometimes put accidental obstacles in their way of achieving said goal. It's amibtious, but as the ancient American Indian proverb is said to state, If you aim your arrow at the sun, it will fly higher and father than if you aim it at the ground.