Does anyone have metrics on the utility of formal Unit Testing?

Question

Does anyone have metrics on the utility of formal Unit Testing? I see a lot of attention being paid to unit testing tools and I was curious why?

I stopped formal unit testing over 5 or 6 years ago and the net gain in productivity seems quite high. I stopped unit testing because I noticed that it never caught anything - let alone anything useful. The type of errors that unit testing detects seem to be preventable by not drinking more than 2 glasses of wine/beer per hour (or 2 joints per hour). Also - unit testing seems to create risk by allowing the developer to think that there is some safeguard to catch their mistakes.

I do test to ensure that the code works as it should, but I do not use any tools. I test based on the changes being made. My production error rate for new code is approximately zero. My error rate for changes to code is about 2-3 bugs per quarter. The above measures are based on 4 production applications that I develop/support.

Resolved:

• Dave Hillier's response was the closest there was to actual metrics.
• I found even those metrics wanting... but who needs metrics when you have such vocal and adamant fans.
• Based on the responses, Formal Unit Testing seems like a religion. You either believe or you don't. If you question it or ask for evidence, you will be persecuted.
• Long Live JEBUS!

Answer 1

I acknowledge your superiority as human being and a coder.

I, however, am a mere moron, and without Python unittest, I would be lost.

I cannot refactor without unit tests, it just takes too much thinking.

I can barely code without unit tests, it's too hard to be absolutely sure I absolutely understand absolutely every nuance.

I unit test because I'm an idiot. Since you don't make mistakes, you clearly don't need to unit test. I salute you.

---

Edit. For me, unit tests aren't about metrics or costs. I don't need any randomized, controlled experiments to show me the value. I cannot work without them. Indeed, I refuse to work without them. In a similar vein, I won't work without a compiler, a text editor, or source code control; I won't work without requirements; I refuse to program without doing design first.

Answer 2

I don't have any metrics to point at, but I think the rise in popularity is because the rest of us have had experience that's the opposite of yours. :)

Answer 3

I do not see unit testing as a replacement for traditional testing, but rather as an extra step to ensure correctness of code. Some particular areas where I find unit testing useful are:

• When refactoring/changing existing code. Unit tests will verify that at least those cases still work as expected. The more tests you have, the more sure you can be that the code changes did not break existing code.
• When submitting bug reports. Having a unit test which exposes a bug is a great way of demonstrating the bug AND knowing when it has been fixed.
• A means of designing interfaces. You have some test code to check the interfaces out with.

Probably a few others I've forgotten about :-P

PS: How do you know you make no bugs? I don't think that I introduce bugs into code I work on, but that certainly doesn't make it so. IMHO, it is naive to think that your code is bug free.

(Regarding unit testing, if you know your code may contain bugs - and I would say most code does - wouldn't you want to use every possible means to catch them?)

Answer 4

With unit tests, I can fix bugs in production code and install the new version within the hour the bug was found and I can be sure that the new version isn't worse than what we had before - because the tests tell me so. It might be better, though.

They give me a lower watermark below which the quality of my code can never sink. They allow me to keep track of the bigger picture and have the tests find the small mistakes that I tend to make. They also allow me to develop in a very relaxed style.

Since I test, I tend to deliver on time, my code quality has improved a lot and the result usually works as expected. Also, I'm much faster since I can cut corners which would be too dangerous to try if I didn't have the tests.

That said, I also don't have any hard numbers nor do I know any source despite the fact that I'm doing unit test and TDD for years. My love for tests is based on pure word of mouth and personal experience.

Answer 5

There are several tools that measure the code coverage with unit tests. They are an essential part together with unit test to ensure the code is not only tested, but completly tested.

Everything else is just pure magic ;)

Answer 6

I've found that unit testing helps me when adding new functionality. In this scenario I used to worry that what I was adding was going to break something in some remote part of the application. But with appropriate unit tests I know whether or not I've broken something the moment I run the tests.

Here's an interesting discussion on the utility of unit tests.

If you don't like unit tests, another concept you might want to look into is Design By Contract. Which basically asserts that if certain input conditions are met then there will be a guaranteed output according to the contract.

Answer 7

If you want to refactor code, by definition you need some way of telling if the changes broke the code. Lacking divine insight, I find that unit testing is a pretty good tool, but ymmv.

Answer 8

Here is some White Paper about Unit Test that might help you:

• White paper #1
• White paper #2
• White paper #3

But, Martin Fowler put it, the anecdotal evidence in support of unit tests and TDD is overwhelming, but you cannot measure productivity.

Unit testing is good because you can change a part and know if somewhere else it has modified something. Some people are "in love" with Unit Testing and should calm theirselve. I believe in Unit Testing but people who try to covert everything are AS dangerous of people who do not unit test.

Answer 9

I've specifically had a lot of gain using Test Driven Development (TDD) with C++ on a huge monolithic server application.

When I'm working on an area of code, I first ensure that that area is covered with tests before I change or write new code.

In this use case I have huge gains in productivity.

• To build and run my test: 10 seconds.
• To build, run and manually test the full server: min 5 minutes.

This means I'm able to iterate an area of code very quickly and only test fully when I need to.

In addition to this I have also utilised integration testing which take longer to build and run, but only exercise the specific piece of functionality I am interested in.

Answer 10

Remember the rise in popularity of 70's and 80's haircuts and clothes... that didn't work out so well for those of us who lived in those decades.

Formal unit testing takes considerable work and effort to maintain. I'd guess that it takes 20-50% of the time it takes to actually develop the software. What I'm asking is for the known price of adding 20-50% overhead to every development effort, is the gain noteworthy and/or proveable.

By not doing formal unit testing, you are forcing the developer to think through the appropriate things to test. The developer takes more ownership of the product.

Formal Unit testing sounds like snake oil juice... Everyone and his brother say it is good, useful, cool, etc., but there has not been a randomized controlled trial to prove that it actually saves time or money. All the responses thus far are subjective testimonies.

What I'd like to know is if there is a software manager who can demonstrate higher productivity (or even higher quality) after the introduction of unit testing.

lol - the facetious rant by S.Lott is the highest ranked response... Given this forum is anonymous (for me anyway), your respect is not what I'm seeking. I'd consider myself barely above mediocre. I've worked with exceptional developers... those guys generally don't even do basic tests of their code - they just know it will work.

Answer 11

Here is a thread that has some research about the TDD approach http://stackoverflow.com/questions/164354/research-on-tdd

http://stackoverflow.com/questions/237000/is-there-hard-evidence-of-the-roi-of-unit-testing

Answer 12

I have some sympathy for what you're saying. My experience is similar in that my new bugs are seldom caught by unit tests. If at all, the unit tests are modified after the bug has been found to ensure that it doesn't reappear.

Where unit tests have helped me is in the design of my (Java) classes. I have often refactored the classes to make them testable (removal of singletons for example) which, I think, has improved the overall design. For me, that's reason enough to use them.

Answer 13

In my experience Unit Testing helped me with the following stuff :

• Now I can give all of my focus to the code block / function / class that I'm writing without worrying about anything else, because I know If I do something stupid or cause a side effect my tests will tell me
• I can refactor stuff by knowing that I'm not breaking stuff
• I'm sure that my app is working as expected
• Before a release I don't check every single functionality manually to confirm that app still works,
• I'm sure my application is always stable in some level

However this is a bit related with me since I'm really bad about "managing multiple stuff at a time" and "focusing". Therefore Unit Testing works for me and literally save my day so many times, where I introduced a new feature and it broke some old functionality.

If this is not the case for you just don't use it. If you are still having the same outcome, performance and quality with the same amount of bugs then that means you don't need unit testing. Or you need to revise your unit testing methodologies.

P.S. Based on your bug rate and what you've said you sound like a genius anyway (assuming these are medium or big projects), so I'd say don't use Unit Tests, it looks like you are doing fine. But for the rest of the world who are not genius such as me I strongly recommend Unit Test because it worked for me.

Answer 14

Dunno about you, but I've just checked in two fixes for two coredumps created by changes in existing code that my unit tests caught. And I just had a major production issue that would have been caught by a unit test if I had more trust in its results (our unittests are a bit more on a functional side than I'd like to admit).

Answer 15

It seems to me that formal Unit Testing with the popular testing tools is pretty much like U.S. airport security.

• It provides the illusion of security
• It makes people 'feel' good
• It's very inconvenient (extremely inconvenient if you got the wrong color skin)
• People will angrily wave their fist at you if you criticise it...
• These same people will be left befuddled when their process fails them and they will jump on the next band wagon...

I think people have different perspectives on software. In my opinion, software is a means of making money (hopefully by providing increased revenues or saving money). I've seen the posts for TDD which is the closest I see as a scientific way to answer the question, but the methodology lacks scientific rigor. None of the articles specified had a baseline or fairly contrasted alternative method.

I guess, the fans of formal unit testing will continue to feel secure in their ways. I will continue to write my spec on a scrap of paper and put in the bowl at the feet of my statue of St. Anthony and say a prayer. Who is to say which way is more effective, but my way sure feels good... Maybe I'll write a white paper about it.

Answer 16

I'm a development manager. For my organization, setting up and migrating to nhibernate involved some setup costs and added to our development time. Some of the developers liked it, some thought it was a waste of time.

There hasn't been a noticeable change in error rates, but perhaps it's too early to tell.

From my perspective, I think it helps junior developers who aren't sure of their work, but for the senior developers, it seems to slow them down - it's one more thing to keep updated. I'm not sure if we'll continue using this, revert back to our old ways (ad hoc unit testing), or let developers make a personal choice.