What is Your Experience with Unit Testing in Practice?

Question

//Yes, Virginia, there is a wikipedia.

What I'm asking here is what you've found unit testing to be in your own practice. It would be helpful to know the pros, cons, obstacles to introducing unit testing, flaws in the unit testing process, value gained by unit testing, and so on that the developers here have observed in their day to day practice.

What we're doing:
I work on a web application w/ an Oracle backend. The application has been in production for 8+ years, and there are ongoing enhancement efforts along with the maintenance work.

My development team does Unit Test Scripts (UTS) that aim to walk a developer through the unit test of a given code package. These UTS are 2-20 page Word documents that describe the purpose of the package, the structure, and elaborate on given modules within a package. The UTS concludes with a section that describes an Application Test of the package.

This doesn't sound like unit testing to me. It sounds a lot more like system integration testing w/ a description of the package. It certainly is not automated.

With our UTS practice in its current state, managing/editing/follwing the documents is not a trivial task. And when enhancements or maintenance work to a given package is needed, there are not strict testing procedures put in place to guarantee outputs are satisfactory. Instead, we have this higher level UTS to follow, which does not come close to guaranteeing correct output in various code traces.

Answer 1

Unit test, for me, has been something of a mixed blessing.

I have been in the situation where I must write unit tests for several years now, and I totally see the advantage for a clean and efficient development cycle, however these are some of the issues I have faced.

• Development may be clean, but when requirements change, you often have to start again at the unit test level. This is, of course, a process issue rather than a TDD one.
• Testing GUI stuff is nigh on impossible OOTB with most tools
• There is a large amount of self-discipline required to make sure you use best practice. It is unfortunate but it's very easy to fall away from test-driven to "test-supported" development. By that I mean starting off with the best of intentions, but not sticking to the idea of writing your test first.

In all, though, I am in favour of using unit testing to improve stability, reliability, brevity and general quality of your code. Regression testing is a huge area for unit tests to come into their own, and I sure hope someone else can explain their views on that a little better than I can. :)

Answer 2

@ZombieSheep: OOTB => "Out of the Box" right?

What tools do you use to prevent "test-supported" and make it more visible as to your code coverage (e.g. NCover)?

Answer 3

If done right, unit testing can be very useful. But to be done right, it really has to be done from the beginning of the project, using some sort of TDD approach. If you write your code first, and then try to write unit tests for it later, you are going to be in for a world of hurt. The code will not likely be very testable, and what you will actually be writing are integration tests, not unit tests.

If you are going to write "true" unit tests, you really need to use TDD principles such as Inversion of Control and Dependency Injection. Without these, you are really just writing integration tests, which are also useful, but they're not unit tests.

Answer 4

If you don't write unit tests from the start, alongside your code, they either do not get written or get written poorly.

The first is the most common occurrence.

Answer 5

First thing I think that I think is important is what bcwood said - understand the difference between a Unit Test and an Integration test. Integration tests are valuable, but typically are harder to write (and change) and run slower than Unit tests. Unit tests should be in total isolation and run quickly. I have a Java web app, written in Spring, that has about 1,000 Unit tests. It takes less than 10 seconds to run.

Next it depends on what kind of application you're testing. If it's something new, you're in luck because you can use the latest and greatest framework that was made with testing in mind. for example, if you're doing a Java web app, go with Spring MVC. Better yet, go with Spring MVC with 2.5 annotations. Even more test-friendly.

Instead of listing the "pros" of Unit testing (they are very well documented elsewhere) here are some of the cons, in my personal experience:

• if you're using a legacy app it might be hard to shoehorn testing in there
• It's hard to get everybody on the team on the same page with testing. some don't see the value, some are lazy, and some just don't get it
• if you thought selling the developers on it was hard, try selling it on the business people who aren't developers but still need to make technology decisions. I don't care how good the team is, there is going to be some ramp-up time for unit testing and the business people may not want to pay for it.
• a lot of teams decide to do integration testing to start, somehow thinking it is more valuable. it's not, I don't think it's less valuable but it's certainly not more valuable. as I said earlier, integration tests are slower to run and harder to write, and in my mind can be very frustrating to somebody new to writing tests. Start with REAL Unit testing, get the hand of it, and then move on to integration tests.
• In my experience having an automated build that does unit testing can be a productivity killer. maybe we just weren't doing it correctly, but what would happen is that several times a day automated tests would be run. if one failed, everybody on the team would receive an email notification. so now it's Office Space where 10 different people are telling you to fix a bug. there's probably a better way to handle this, but in my experience it wasn't very valuable.

Answer 6

My first attempts on unit testing and TDD was some 9-10 years ago on a VB6 backed web project. The tool support was poor, no automatic refactoring, and my test code totally sucked. The test code was hard to understand, brittle, and hard to change.

However, I wrote fewer bugs, did almost no debugging and delivered on time and budget. In my opinion, the time not spent in the debugger probably paid the learning time.

Answer 7

Our organisation is in a similar position to Gary's, we have a java code base of around 3000 classes that has been in production for years. Unit and integration (and regression) testing are relatively new concepts (within the last 2 1/2 years).

It definitely sounds like Gary is describing an integration testing system, and almost could be fit tables.

Regarding existing code:

To try and write unit or integration tests for the complete code base would be a nightmare, not to mention a massive drain on already stretched dev resources. And for those suggesting it, re-writing is not an option!

The general rule that the developers have is that all new code should be written with an accompanying unit test, and integration if applicable.

If existing code is being edited, and is too complex to be tested, it is a candidate to be refactored into a form that can be tested. We hold short weekly meetings to look at some of these.

Automation:

We use Pulse and I personally use TeamCity for continuous build servers, which run unit and integration tests on each check-in, as well as nightly Selenium regression tests. The build server is a key element because it provides automation of the tests - and at any time we can see the state of the build, the number of tests being run, and the test coverage (which is gradually improving!)

Pros/Cons:

• Writing unit test for new code is fairly trivial, if you keep the test simple and atomic, and are thinking about the test, if not writing it, before you write your production code.

• Attempting to write tests for existing code is a massive time-sink, and efforts to refactor the code can be lengthy and complicated.

• Convincing lazy programmers to write tests for their code can be even harder, even though any programmer worth their salt should realise that unit tests are as essential as version control.

• Writing code that is unit-testable means it is also maintainable. If you're dealing changing requirements from customers and new laws, you will be improving the warranty turn-around to implement them.

• An automation strategy is essential to making the unit-test investment worthwhile. If you're getting a near-instant response to your code changes, the build is going to be fixed much more quickly!

---

I think that our tests have picked up a large number of minor bugs that would otherwise hold up our release schedule, or be reported by our customers, since our code is large enough that some things may go for months without being used after the dev has finished with it.
I don't know if we've avoided major disasters because of them, but we can at least be more confident that our code does what we expect it to.

Answer 8

I have been on some projects with heavy unit testing (on the order of an entire team devoted to nothing else but writing and running unit tests) and some with lighter unit tests.

Over the years, I have come to feel that unit tests to be effective must be thought of as scaffolding. That is to say, over time as code is built up you spend more and more work working around the unit tests or repairing them - in some cases I think you must retire tests that are creating more work than return, that are returning more false positives than helpful flags when you make a mistake.

I am not sure if any of the current unit testing frameworks support easy retiring or analyses of unit tests to determine which are no longer necessary, but if there are not tools to help and some point someone has to be willing to go in and weed.

Along the same line of thinking I think TDD (test driven development) is great at first, but like a first stage booster rocket must at some point be ejected so less purely test driven system can move the project forward.

Answer 9

My experience is that it is probably not worth doing unless you are starting on a new codebase. I tried to add it in after the fact to a legacy C++ codebase using Boost.Test and found that it was too difficult to do. Code needs to be written in a way that is implicitly testable, legacy code is probably unlikely to be written in such a way.

You really need to design the app from the ground up with testability in mind. TDD is a good method of doing this (I guess). Otherwise, you will find that the methods and objects that need to be stubbed in or out are too tightly woven into the fabric of the app to easily remove later on.