I always learned that doing maximum code coverage whit unit tests is good. I also hear developers from big companies such as Microsoft saying that they write more lines of testing code than the executable code itself.

Now, is it really great? Doesn't it seem sometimes like a complete loss of time which has an only effect to making maintenance more difficult?

For example, let's say I have a method DisplayBooks() which populates a list of books from a database. The product requirements tell that if there are more than one hundred books in the store, only one hundred must be displayed.

So, with TDD,

  1. I will start by making an unit test BooksLimit() which will save two hundred books in the database, call DisplayBooks(), and do an Assert.AreEqual(100, DisplayedBooks.Count).
  2. Then I will test if it fails,
  3. Then I'll change DisplayBooks() by setting the limit of results to 100, and
  4. Finally I will rerun the test to see if it succeeds.

Well, isn't it much more easier to go directly to the third step, and do never make BooksLimit() unit test at all? And isn't it more Agile, when requirements will change from 100 to 200 books limit, to change only one character, instead of changing tests, running tests to check if it fails, changing code and running tests again to check if it succeeds?

Note: lets assume that the code is fully documented. Otherwise, some may say, and they would be right, that doing full unit tests will help to understand code which lacks documentation. In fact, having a BooksLimit() unit test will show very clearly that there is a maximum number of books to display, and that this maximum number is 100. Stepping into the non-unit-tests code would be much more difficult, since such limit may be implemented though for (int bookIndex = 0; bookIndex < 100; ... or foreach ... if (count >= 100) break;.

+5  A: 

When looking at an isolated problem, you're completely right. But unit tests are about covering all the intentions you have for a certain piece of code.

Basically, the unit tests formulate your intentions. With a growing number of intentions, the behavior of the code to be tested can always be checked against all intentions made so far. Whenever a change is made, you can prove that there is no side-effect which breaks existing intentions. Newly found bugs are nothing else but an (implicit) intention which is not held by the code, so that you formulate your intention as new test (which fails at first) and the fix it.

For one-time code, unit tests are indeed not worth the effort because no major changes are expected. However, for any block of code which is to be maintained or which serves as component for other code, warranting that all intentions are held for any new version is worth a lot (in terms of less effort for manually trying to check for side effects).

The tipping point where unit tests actually save you time and therefore money depends on the complexity of the code, but there always is a tipping point which usually is reached after only few iterations of changes. Also, last but not least it allows you to ship fixes and changes much faster without compromising the quality of your product.

Maybe also worth a read:
And the always good:
@nico, thanks for the link. However, it seems to me that it quite misses the point of unit tests. You cannot unit test a complex application or even a UI. But you're not supposed to do that using unit tests. They should *really* not have *any* external dependency or indeterministic part in them, or they aren't conceptually unit tests. That said, I agree with the author of the article that you should write quality code in a defensive matter, and that you cannot just code some unit tests and think that the application is tested. But unit tests are perfect for keeping intentions valid on changes.
@Lucero: of course. I guess the author of that post is being a bit too "rough" (on purpose I guess). He's saying that you should not JUST rely on them, and I think we agree here.
+3  A: 

There is no exlpicit relation between code coverage and good software. You can easily imagine piece of code that has 100%(or close) code coverage and it still contains a lot of bugs. (Which does not mean that tests are bad!)

Your question about agility of 'no test at all' approach is a good one only for short perspective (which means it is most likely not good if you plan to build your program for longer time). I know from my experience that such simple tests are very useful when your project gets bigger and bigger and at some stage you need to make significant changes. This can be a momment when you'll say to yourself 'It was a good decision to spend some extra minutes to write that tiny test that spotted bug I just introduced!".

I was a big fan of code coverage recently but now it turned (luckilly) to something like 'problems coverage' approach. It means that your tests should cover all problems and bugs that were spotted not just 'lines of code'. There is no need to do a 'code coverage race'.

I understand 'Agile' word in terms of number tests as 'number of tests that helps me build good software and not waste time to write unnecessary piece of code' rather than '100% coverage' or 'no tests at all'. It's very subjective and it based on your experience, team, technology and many others factors.

The psychological side effect of '100% code coverage' is that you may think that your code has no bugs, which never is true:)

Lukasz Dziedzia
I don't completely agree here. 100% code coverage does not mean no bugs, but if there is code which is never used by your tests you either don't need that code (-> throw it away) or you are missing some of the test cases which do something with this code. Of course, there may be simple things such as overloads of a function with default values which is not covered by tests, but in general, you should try to cover all the intentions of all your code.
Well, I agree with you:). Maybe you didn't get my point (= I didn't make my point clear). I just wanted to point out that relaying only on code coverage and forcing to have 100% 'whatever it takes' is a road to nowhere(you give some examples where this is valid). Saying that 'problems' should be covered rather than 'lines of code' is just some kind of reversed thinking of your software but at the end this also leads to have as many tests as necessary. My intention was to point out that keeping high code coverage chould be used wisely not mechanically.
Lukasz Dziedzia
I also totally agree that not covered code should be considered as code to be removed and this is a great point here in this discussion! If you have code that is not covered with tests you can analyze it and decide why this code is not covered and do one of 2 things: remove it from your app or write some tests. I believe many of programmers in such situation just write tests to cover uncovered code because of '100% coverage religion' without 'listenint to their software'. And I wanted to warn about such approach.
Lukasz Dziedzia
@Dzida, thanks for the clarification. Indeed I think that it could be put that way: 100% code-coverage is not the goal of unit test but rather a side-effect of them. Still unit tests should yield a coverage near 100% for "infrastructure"-like code (e.g. ignoring code such as for UI views), and code not used should indeed be considered for removal (thanks to SCM you can sill get it back if you later find out that it was required after all).
Exactly! This is a great one:"Indeed I think that 100% code-coverage is not the goal of unit test but rather a side-effect of them". Thanks for a nice discussion:). I will look to my answer again and try to rewrite it to be more precise (I am not an English speaking person natively).
Lukasz Dziedzia
@Dzida, thanks for the feedback. And actually I'm not a native English speaker either, I'm about 1tkm southwest on the same continent... ;)
+18  A: 

Well, isn't it much more easier to go directly to the third step, and do never make BooksLimit() unit test at all?

Yes... If you don't spend any time writing tests, you'll spend less time writing tests. Your project might take longer overall, because you'll spend a lot of time debugging, but maybe that's easier to explain to your manager? If that's the case... get a new job! Testing is crucial to improving your confidence in your software.

Unittesting gives the most value when you have a lot of code. It's easy to debug a simple homework assignment using a few classes without unittesting. Once you get out in the world, and you're working in codebases of millions of lines - you're gonna need it. You simply can't single step your debugger through everything. You simply can't understand everything. You need to know that the classes you're depending on work. You need to know if someone says "I'm just gonna make this change to the behavior... because I need it", but they've forgotten that there's two hundred other uses that depend on that behavior. Unittesting helps prevent that.

With regard to making maintenance harder: NO WAY! I can't capitalize that enough.

If you're the only person that ever worked on your project, then yes, you might think that. But that's crazy talk! Try to get up to speed on a 30k line project without unittests. Try to add features that require significant changes to code without unittests. There's no confidence that you're not breaking implicit assumptions made by the other engineers. For a maintainer (or new developer on an existing project) unittests are key. I've leaned on unittests for documentation, for behavior, for assumptions, for telling me when I've broken something (that I thought was unrelated). Sometimes a poorly written API has poorly written tests and can be a nightmare to change, because the tests suck up all your time. Eventually you're going to want to refactor this code and fix that, but your users will thank you for that too - your API will be far easier to use because of it.

A note on coverage:

To me, it's not about 100% test coverage. 100% coverage doesn't find all the bugs, consider a function with two if statements:

// Will return a number less than or equal to 3
int Bar(bool cond1, bool cond2) {
  int b;
  if (cond1) {
  } else {

  if (cond2) {
  } else {

Now consider I write a test that tests:

EXPECT_EQ(3, Bar(true, true));
EXPECT_EQ(3, Bar(false, false));

That's 100% coverage. That's also a function that doesn't meet the contract - Bar(false, true); fails, because it returns 4. So "complete coverage" is not the end goal.

Honestly, I would skip tests for BooksLimit(). It returns a constant, so it probably isn't worth the time to write them (and it should be tested when writing DisplayBooks()). I might be sad when someone decides to (incorrectly) calculate that limit from the shelf size, and it no longer satisfies our requirements. I've been burned by "not worth testing" before. Last year I wrote some code that I said to my coworker: "This class is mostly data, it doesn't need to be tested". It had a method. It had a bug. It went to production. It paged us in the middle of the night. I felt stupid. So I wrote the tests. And then I pondered long and hard about what code constitutes "not worth testing". There isn't much.

So, yes, you can skip some tests. 100% test coverage is great, but it doesn't magically mean your software is perfect. It all comes down to confidence in the face of change.

If I put class A, class B and class C together, and I find something that doesn't work, do I want to spend time debugging all three? No. I want to know that A and B already met their contracts (via unittests) and my new code in class C is probably broken. So I unittest it. How do I even know it's broken, if I don't unittest? By clicking some buttons and trying the new code? That's good, but not sufficient. Once your program scales up, it'll be impossible to rerun all your manual tests to check that everything works right. That's why people who unittest usually automate running their tests too. Tell me "Pass" or "Fail", don't tell me "the output is ...".

OK, gonna go write some more tests...

Not everything can be tested with unit tests. And debugging is not just about writing unit tests. Trying the software, finding UI bugs, and **human driven** beta testing is also essential to find bugs. The question is: is it more prioritary to correct that graphic glitch that EVERY user will see (and be annoyed by) or that remote bug that you could find with a unit test but that will only be seen once in a million times when using the software? Is it worth spending more time writing tests then coding when you could spend that time using your software to understand WHERE to improve it?
@Nico : That's a good point. I neglected to talk about other types of testing (mostly because the question was directed at unittesting). System testing has proved (to me) to be indispensable. I mostly write infrastructure, which is providing an implementation of a contract, so unittesting works well. I don't have a UI to test, but if I did, testing it would be important.
@Stephen: Of course I'm not saying that unit tests are always useless. I'm just saying that one should not just rely on them to say that their software is "bug free (TM)" :)
@nico, unit tests should test isolated functionality, they are neither system nor integration tests and don't replace those. However, by verifying intentions on code units, they ensure stable overall behavior of the code, especially when changes/enhancements are made, and therefore also reduce the follow-up costs (in terms of roundtrip time and effort) of the more complicated and therefore more "expensive" complex tests.
*"For a maintainer (or new developer on an existing project) unittests are key."* I understand. So let's see, in my example, if a new developer arrives and tries to modify `DisplayBooks()` method, he can simply be unaware of 100 books limit requirement. So, changing something, he maybe would by mistake remove dozens of lines, including the one which sets the limit. With unit testing, he will notice immediately that he has broken something. Without, the product will fail to behave as expected once released.
@MainMa : exactly. accuracy of documentation erodes, and people just don't read it - who can blame them? The "truth" is in the code! Unittests are a way of checking those requirements. Lately, as the "new guy", a lot of my questions are of the form "I change the behavior from X to Y and no unittests fail. Is behavior X important? Will anyone care?". If the answer is "yes", it's insufficiently tested, so we add some.
+2  A: 

100% unit test coverage is generally a code smell, a sign that someone has come over all OCD over the green bar in the coverage tool, instead of doing something more useful.

Somewhere around 85% is the sweet spot, where a test failing more often that not indicates an actual or potential problem, rather than simply being an inevitable consequence of any textual change not inside comment markers. You are not documenting any useful assumptions about the code if your assumptions are 'the code is what it is, and if it was in any way different it would be something else'. That's a problem solved by a comment-aware checksum tool, not a unit test.

I wish there was some tool that would let you specify the target coverage. And then if you accidentally go over it, show things in yellow/orange/red to push you towards deleting some of the spurious extra tests.

+1  A: 

First 100% is hard to get especially on big projects ! and even if you do when a block of code is covered it doesn't mean that it is doing what it is supposed to unless your test are asserting every possible input and output (which is Almost impossible).

So i wouldn't consider a piece of software to be good simply because it has 100% code coverage but code coverage still a good thing to have.

Well, isn't it much more easier to go directly to the third step, and do never make BooksLimit() unit test at all?

well having that test there makes you pretty confident that if someone changes the code and the test fails you will notice that something is wrong with the new code therefore you avoid any potencial bug in your application

+1  A: 

I agree with @soru, 100% test coverage is not a rational goal.

I do not believe that any tool or metric can exist that can tell you the "right" amount of coverage. When I was in grad school, my Thesis advisor's work was on designing test coverage for "mutated" code. He's take a suite of tests, and then run an automated program to make errors in the source code under test. The idea was that the mutated code contained errors that would be found in the real world, and thus a test suite that found the highest percentage of broken code was the winner.

While his thesis was accepted, and he is now a Professor at a major engineering school, he did not find either:

1) a magic number of test coverage that is optimal 2) any suite that could find 100% of the errors.

Note, the goal is to find 100% of the errors, not to find 100% coverage.

Whether @soru's 85% is right or not is a subject for discussion. I have no means to assess if a better number would be 80% or 90% or anything else. But as a working assessment, 85% feels about right to me.


When the client decides to change the limit to 200, good luck finding bugs related to that seemingly trivial test. Specially, when you have other 100 variables in your code, and there are other 5 developers working on code that relies on that tiny piece of information.

My point: if it's valuable to the business value (or, if you dislike the name, to the very important core of the project), test it. Only discard when there is no possible (or cheap) way of testing it, like UI or user interaction, or when you are sure the impact of not writing that test is minimal. This holds truer for projects with vague, or quickly changing requirements [as I painfully discovered].

For the other example you present, the recommended is to test boundary values. So you can limit your tests to only four values: 0, some magical number between 0 and BooksLimit, BooksLimit, and some number higher.

And, as other people said, make tests, but be 100% positive something else can fail.