views:

406

answers:

9

I'm a senior engineer working in a team of four others on a home-grown content management application that drives a large US pro sports web site. We have embarked upon this project some two years ago and chose Java as our platform, though my question is not Java-specific. Since we started, there has been some churn in our ranks. Each one of us has a significant degree of latitude in deciding on implementation details, although important decisions are made by consensus.

Ours is a relatively young project, however we are already at a point when no single developer knows everything about the app. The primary reasons for that are our quick pace of development, most of which occurs in a crunch leading up to our sport's season opener; and the fact that our test coverage is essentially 0.

We all understand the theoretical benefits of TDD and agree in principle that the methodology would have improved our lives and code quality if we had started out and stuck with it through the years. This never took hold, and now we're in charge of an untested codebase that still requires a lot of expansion and is actively used in production and relied upon by the corporate structure.

Faced with this situation, I see only two possible solutions: (1) retroactively write tests for existing code, or (2) re-write as much of the app as is practical while fanatically adhering to TDD principles. I perceive (1) as by and large not practical because we have a hellish dependency graph within the project. Almost none of our components can be tested in isolation; we don't know all the use cases; and the uses cases will likely change during the testing push due to business requirements or as a reaction to unforeseen issues. For these reasons, we can't really be sure that our tests will turn out to be high quality once we're done. There's a risk of leading the team into a false sense of security whereby subtle bugs will creep in without anyone noticing. Given the bleak prospects with regards to ROI, it would be hard for myself or our team lead to justify this endeavor to management.

Method (2) is more attractive as we'll be following the test-first principle, thus producing code that's almost 100% covered right off the bat. Even if the initial effort results in islands of covered code at first, this will provide us with a significant beachhead on the way to project-wide coverage and help decouple and isolate the various components.

The downside in both cases is that our team's business-wise productivity could either slow down significantly or evaporate entirely during any testing push. We can not afford to do this during the business-driven crunch, although it's followed by a relative lull which we could exploit for our purposes.

In addition to choosing the right approach (either (1), (2), or another as of yet unknown solution), I need help answering the following question: How can my team ensure that our effort isn't wasted in the long run by unmaintained tests and/or failure to write new ones as business requirements roll on? I'm open to a wide range of suggestions here, whether they involve carrots or sticks.

In any event, thanks for reading about this self-inflicted plight.

+8  A: 

I need help answering the following question: How can my team ensure that our effort isn't wasted in the long run by unmaintained tests and/or failure to write new ones as business requirements roll on?

Make sure that your build process executes the tests on every build and fails the build if there are failures.

Do you use Continuous Integration? Hudson is a great tool for this. It can keep a graph of # of tests, # of failures, test coverage, etc., for every build over the lifetime of your project. This will help you keep an easy eye on when your coverage % is declining.

As you mentioned it can be pretty hard to retrofit unit testing into an existing project, let alone TDD. I wish you luck on this effort!

Update: I also want to point out that 100% test coverage isn't really a great goal, it has diminishing returns as you try to go from ~80% or ~90%. To get those last few percentage points you need to start simulating every possible branch in your code. Your team will start spending time simulating scenarios that either can't happen in real life ("this stream won't actually throw an IOException when I close it but I need to get this branch covered!") or has no real value in your testing. I caught someone on my team verifying that if (foo == null) throw new NullPointerException(...); as the first line of a method actually threw an exception when the value was null.

Much better to spend your time testing the code that actually matters than becoming obsessive-compulsive about making every last line show up as green in Emma or Cobertura.

matt b
Couldn't agree more. The main reason people fail to adopt practices that involve automated testing is because they lack a continuous integration system to, well, keep the tests automated and not a burden to the developers.
Mike
We started out with CruiseControl as our CI platform, but that fell by the wayside about as quickly as our initial attempts at unit testing. CI will have to make a comeback if we're to succeed in doing TDD effectively. Thanks for reminding me of this.
Max A.
for Java projects, I strongly recommend Sonar (http://sonar.codehaus.org/)
chburd
Hudson is dead simple to set up and I would recommend setting it up to build your project even if you have zero tests. At the very least you'll get warnings of when someone checks in un-compiling code. You can also then tag each build by number in your SCM, very easily add on plugins to generate findbugs and checkstyle warnings by build, etc.
matt b
Finally, Hudson can build a large number of non-Java projects as well, such as .NET, Ruby, Python, C/C++, etc
matt b
You're raising a good point, OCD behavior won't get me anywhere. I should rephrase my statement: I'd like to aim for 100% coverage of all feasible use cases. On some level though, I feel like code shouldn't be written to support off-the-wall ridiculous use cases to begin with.
Max A.
+1  A: 

I've run into this in the past with a few teams. When I've been the tech lead on the project this is the way I approached it. First, I had the developer write out what the test case would be (via a design document or such). That way not only do we have the test case but it parrots back to me their understanding of the change so I can verify that they know what they're doing.

After that for non-GUI tests I have the developer use JUnit or something like that to make sure it's good. Since the test case is already defined this doesn't take too long.

Depending on the complexity of the regressive test case it could take a little longer but after explaining the better maintainability, fewer bugs because of changing existing code with these kind of tests and such.....I've usually been able to push it through.

Hope this helps.

SOA Nerd
+1 for tests as a communication tool.
Carl Manaster
+2  A: 

I think that you need to flip around part of the question and the related arguments. How can you ensure that adhering to TDD will not result in wasted effort?

You can't, but the same is true for any development process.

I've always found it slightly paradoxical that we are always challenged to prove the cash benefit of TDD and related agile disciplines, when at the same time traditional waterfall processes more often than not result in

  • missed deadlines
  • blown budgets
  • death marches
  • developer burn-out
  • buggy software
  • unsatisfied customers

TDD and other agile methodologies attempt to address these issues, but obviously introduce some new concerns.

In any case I'd like to recommend the following books that may answer some of your questions in greater detail:

Mark Seemann
+1: Crunch coding is clearly more expensive than TDD. It feels better, but the long-term consequences of unmaintainable and untestable code clearly outweigh the immediate gratification of having something that appears to work.
S.Lott
Who said anything about crunches feeling better? Crunches make this job a ridiculous roller coaster ride that puts us through a rush once or twice a year. The remaining months are spent decompressing, almost to the point where nothing gets done because everyone's on vacation.
Max A.
+1 for WELC (particularly; it's a good answer even without that - but read the book!)
Carl Manaster
+8  A: 

I would recommend that all new code and bug fixes required a unit test. Period.

Then, I would go back (before refactoring the older code) and write unit tests for those pieces of code that are the most business critical, has the most bugs, and least understood.

Kris Krause
+1: Incremental change is easier. Wholesale change is hard.
S.Lott
+9  A: 

"The downside in both cases is that our team's business-wise productivity could either slow down significantly or evaporate entirely during any testing push."

This is a common misinterpretation of the facts. Right now you have code you don't like and struggle to maintain. "hellish dependency graph", etc.

So, the "crunch" development you've been doing has lead to expensive rework. Rework so expensive you don't dare attempt it. That says that your crunch development isn't very effective. It appears cheap at the time, but in retrospect, you note that you're really throwing development money away because you've created problematic, expensive software instead of creating good software.

TDD can change this so that you aren't producing crunch software that's expensive to maintain. It can't fix everything, but it can make it clear that changing your focus from "crunch" can produce better software that's less expensive in the long run.

From your description, some (or all) of your current code base is a liability, not an asset. Now think what TDD (or any discipline) will do to reduce the cost of that liability. The question of "productivity" doesn't apply when you're producing a liability.

The Golden Rule of TDD: If you stop creating code that's a liability, the organization has a positive ROI.

Be careful of asking how to keep up your current pace of productivity. Some of that "productivity" is producing cost with no value.

"Almost none of our components can be tested in isolation; we don't know all the use cases"

Correct. Retro-fitting unit tests to an existing code base is really hard.

"There's a risk of leading the team into a false sense of security whereby subtle bugs will creep in without anyone noticing"

False. There's no "false sense of security". Everyone knows the testing is rocky at best.

Further, now you have horrifying bugs. You have problems so bad you don't even know what they are, because you have no test coverage.

Trading up to a few subtle bugs is still a huge improvement over code you cannot test. I'll take subtle bugs over unknown bugs any day.

"Method (2) is more attractive"

Yes. But.

Your previous testing efforts were subverted by a culture that rewards crunch programming.

Has anything changed? I doubt it. Your culture still rewards crunch programming. Your testing initiative may still get subverted.

You should look at a middle ground. You can't be expected to "fanatically adhering to TDD principles" overnight. That takes time, and a significant cultural change.

What you need to do is break your applications into pieces.

Consider, for example, the Model - Services - View tiers.

You have core application model (persistent things, core classes, etc.) that requires extensive, rigorous trustworthy testing.

You have application services that require some testing, but are subject to "the uses cases will likely change during the testing push due to business requirements or as a reaction to unforeseen issues". Test as much as you can, but don't run afoul of the imperative to ship stuff on time for the next season.

You have view/presentation stuff that needs some testing, but isn't core processing. It's just presentation. It will change constantly as people want different options, views, reports, analysis, RIA, GUI, glitz, and sizzle.

S.Lott
This is really great advice
matt b
+1 Good answer.
Mark Seemann
Okay, so what you're really saying is that we need to strike a balance between testing what we have to and leaving alone what we don't have to test. This makes total sense and is a good idea, but will require a fair amount of refactoring. For example, our core DAO code merges objects into the Hibernate session and validates them at the same time, in the same method. REST servlets are busy parsing requests for actions and unmarshalling objects at the same time. And so on.
Max A.
+1 Spot on, as usual.
duffymo
@Max A.: Right. You have a mess. You need to refactor. TDD can't retroactively undo the mistakes you made. You have hard, hard work to do. And you have a culture that rewards crunch development. You created the problem one bad decision at a time. You can only fix the problem through a similar incremental one-good-decision-at-a-time process.
S.Lott
+4  A: 

I perceive (1) as by and large not practical because we have a hellish dependency graph within the project. Almost none of our components can be tested in isolation; we don't know all the use cases;

This is your real problem.

You can start by writing integration tests that essentially automate your testing process. You get value out of this right out of the gate. What you need is a safety net for refactoring that will give you a hint whenever you've broken the code. Take the widest swath of transactions you can to exercise the app, automate the process of pumping them through and comparing expected to actual.

Once you have that in place, start trying to break some of that dependency graph so you can isolate pieces to test.

Whenever you have a bug to fix, write a unit test for it that replicates the error. Then fix the code and re-run the test. You should see the test pass successfully. Make this your standard bug tracking and fixing procedure. That list of tests will grow like interest in a bank account.

I agree with the CI recommendation. Add code coverage metrics to either Hudson or Cruise Control.

It goes without saying that you're using Subversion, Git, or another source code management system, right?

It's a long process, but worth it in the end.

duffymo
Re: integration tests. I wrote a framework that captured the most common use cases (in terms of data in, data out) and massaged the app externally, the way our consumers and producers would. This worked for a while but still would only tell us that *something* was broken.
Max A.
Agreed - it's a very coarse safety net, but it's better than nothing. Put that in place and keep adding to the unit tests as you write new code.
duffymo
+2  A: 

Retro-fitting tests will allow you to improve your design and find bugs. Trying to rewrite big swaths of your application will just result in you missing your deadline and ending up with a bunch of half-working pieces of functionality. Major rewrites almost never work. Go for option (1).

Oh and just having hudson running and making it complain when you break the tests (through e-mail) is probably enough to get you in the spirit of things.

wds
+1  A: 

Along with some of the excellent suggestions already, I will chime in with two points.

  • Start writing test cases for all new code. This is a must and you need to make it part of your culture.
  • Start writing tests to replicate any bugs that you find in existing code. Before fixing any bug that you find in your existing code base, write a reproducible test case for that bug. This will at the very least allow you to start introducing test cases against areas that are known issues in your code. Although the ideal would be to write tests against all your existing code, this is seldom feasible, so at least address known issues.

In addition, I definitely agree with the Hudson for CI suggestions. In addition, if you aren't doing it already, do peer reviews for all checked in code as well. This does not have to be a drawn out formal process.

For example, we simply have to assign any completed task (via JIRA) to a Code Review status when the developer is done. The developer will choose another developer to assign this task to, and we shoot for a 48 hour turnaround time on code reviews. The task is then marked resolved by the reviewer, not the developer. This gives some extra confidence to the quality of the code, the test coverage and design. Tasks are never resolved by the person who implemented them. An added benefit is that it exposes the code to others so there is at least some knowledge transfer inherent in the process.

Robin
The code review idea is great. If nothing else, it will reduce or eliminate the element of surprise when someone has merged a branch that swaps out the authentication system or whatnot.
Max A.
+1  A: 

This may sound odd, but what actual problems do you have now? You don't say what problems the business is experiencing.

Sure, if I joined your team, I'd be doing TDD as much as I could, I've been committed to practising TDD for many years now, but I've also seen ugly code bases and wanted to clean them up, but not understood enough to be sure any source code changes I wanted to do to improve testability were actual refactorings.

You sound like you have a good grasp of the reality of the situation, but can you prove that doing TDD would improve the situation? Have you got any metrics that highlight your problems? If you have then you can use better practises to improve them.

OK, after all that, my practical advice would be to start by using your relative lull to do whatever you can, including: documenting test cases, writing automated tests, refactoring to reduce dependencies, writing new code with TDD, training up your programmers to be effective with TDD, etc.

quamrana
The one metric that's worth focusing on in my case is the code base's fragility. Each time someone makes changes to the core code there's high risk of the entire application breaking in a subtle way. Obviously, there's no way for the originating developer to test every conceivable use case, which is what tests are for. Now, when I say TDD, I mean "test before you code", where the tests might be unit or integration tests, or anything in between.
Max A.
To continue on the theme: a fragile code base means boat loads of dev time lost debugging weird problems that arise from someone else's work. The issues aren't always obvious even to the person who (inadvertently) caused them. If I had to give a number, I'd say that anywhere between 60% and 80% of my supposedly 40-hour work week is taken up by this useless debugging that could have been avoided if tests were in place.
Max A.