views:

4270

answers:

35

What do I lose by adopting test driven design?

List only negatives; do not list benefits written in a negative form.

A: 

You lose the ability to write code without a clear knowledge of the success and fail conditions.

Thunder3
Is that really a downside?
Thomas Owens
Second Thomas here.. TBH, that would normally come from a solid spec, and when was the last time you got one of those? I find being able to "code for the moment" enables you to be much more dynamic so you dont NEED absolute knowledge of success/fail.. Just my thoughts..
Rob Cooper
That's a "benefit written in a negative form". Programming is trade-offs; what do you give up when you get knowledge of success and failure? The original poster asked for disadvantages.
Ron Romero
+2  A: 

You lose a lot of time spent writing tests. Of course, this might be saved by the end of the project by catching bugs faster.

Joel Coehoorn
Is this really a negative, or sly way of stating a positive.
IanL
A: 

TDD requires a certain organization for your code. This might be inefficient or difficult to read. Or even architecturally wrong; for example, since private methods cannot be called outside a class, you have to make methods non-private to make them testable, which is just wrong.

When code changes, you have to change the tests as well. With refactoring this can be a lot of extra work.

Jason Cohen
All private methods should be tested through the public methods that would exist anyway.
Garry Shutler
This is not possible with all classes. Sometimes you don't want to mock up all dependencies etc. and you just wish to test a utility method.
Jason Cohen
+1, so true. Add to this the requirement to sometimes add getters/setters to private fields just to be able to set up and read state correctly for a unit test, even though the state should be private to the class.
erikkallen
+6  A: 

Well, and this stretching, you need to debug your tests. Also, there is a certain cost in time for writing the tests, though most people agree that it's an up-front investment that pays off over the lifetime of the application in both time saved debugging and in stability.

The biggest problem I've personally had with it, though, is getting up the discipline to actually write the tests. In a team, especially an established team, it can be hard to convince them that the time spent is worthwhile.

Tim Sullivan
+2  A: 

I second the answer about initial development time. You also lose the ability to confortably work without the safety of tests. I've also been described as a TDD nutbar, so you could lose a few friends ;)

Chris Canal
A: 

It's percieved as slower. Long term that's not true in terms of the grief it will save you down the road, but you'll end up writing more code so arguably you're spending time on "testing not coding". It's a flawed argument, but you did ask!

Marc
+23  A: 

When you get to the point where you have a large number of tests, changing the system might require re-writing some or all of your tests, depending on which ones got invalidated by the changes. This could turn a relatively quick modification into a very time-consuming one.

Also, you might start making design decisions based more on TDD than on actually good design prinicipals. Whereas you may have had a very simple, easy solution that is impossible to test the way TDD demands, you now have a much more complex system that is actually more prone to mistakes.

Eric Z Beard
Definately can be a problem, however I find I am seeing a noticable difference in how much I am affected by this. All comes down to "writing testable code" I guess.
Rob Cooper
Doesn't a unit test by definition only test the smallest unit of code possible. A function, method, etc..Since refactorings are supposed to be incremental in a TDD approach, the total % of your test codebase that breaks should be small for any given refactoring.Right?
Allain Lalonde
@Allain, that's the theory, in practice things are always more complicated, especially when working with a team. There's nothing magical about test code that automatically makes it better written or better designed. It's a classic "who watches the watchers" problem.
Wedge
I completely agree that the more tests you have the more potential test refactor you'll have to do when making system changes. I'm confused about the statement regarding making design decisions based on TDD? That's one of the major benefits of TDD, right? It drives out design by nature?
Scott Saad
Scott, the example I usually give is a SqlDataSource embedded in an ASPX page. You can't automate a test for that. It's simple and gets the job done, with just 1 file. The testable component is MSFT's SqlDataSource object, and that's done for us already. No need for us to do more.
Eric Z Beard
A: 

It can be hard and time consuming writing tests for "random" data like XML-feeds and databases (not that hard). I've spent some time lately working with weather data feeds. It's quite confusing writing tests for that, at least as i don't have too much experience with TDD.

Vargen
This is a common problem. I tend to mock these with hard coded objects, then test the database seperately. Your business layer should then only work with static data, you DAL will then be tested in a controlled environment (where you can script the data into it etc.)
Rob Cooper
+8  A: 

I think the biggest problem for me is the HUGE loss of time it takes "getting in to it". I am still very much at the beginning of my journey with TDD (See my blog for updates my testing adventures if you are interested) and I have literally spent hours getting started.

It takes a long time to get your brain into "testing mode" and writing "testable code" is a skill in itself.

TBH, I respectfully disagree with Jason Cohen's comments on making private methods public, that's not what it is about. I have made no more public methods in my new way of working than before. It does, however involve architectural changes and allowing for you to "hot plug" modules of code to make everything else easier to test. You should not be making the internals of your code more accessible to do this. Otherwise we are back to square one with everything being public, where is the encapsulation in that?

So, (IMO) in a nutshell:

  • The amount of time taken to think (i.e. actually grok'ing testing).
  • The new knowledge required of knowing how to write testable code.
  • Understanding the architectural changes required to make code testable.
  • Increasing your skill of "TDD-Coder" while trying to improve all the other skills required for our glorious programming craft :)
  • Organising your code base to include test code without screwing your production code.

PS: If you would like links to positives, I have asked and answered several questions on it, check out my profile.

Rob Cooper
A: 

You lose the ability to make incremental changes (code refactorings) and still feel warm and fuzzy that the code does what it is supposed to. You lose practically free and painless motivation to structure your code with minimal explicit dependencies. IOW, you'll be able to embed lots of dependencies without noticing. Were you to use TDD the dependencies would show up as pain/smell when writing the tests.

A: 

You will lose large classes with multiple responsibilities. You will also likely lose large methods with multiple responsibilities. You may lose some ability to refactor, but you will also lose some of the need to refactor.

Jason Cohen said something like: TDD requires a certain organization for your code. This might be architecturally wrong; for example, since private methods cannot be called outside a class, you have to make methods non-private to make them testable.

I say this indicates a missed abstraction -- if the private code really needs to be tested, it should probably be in a separate class.

Dave Mann

+2  A: 

TDD requires you to plan out how your classes will operate before you write code to pass those tests. This is both a plus and a minus.

I find it hard to write tests in a "vacuum" --before any code has been written. In my experience I tend to trip over my tests whenever I inevitably think of something while writing my classes that I forgot while writing my initial tests. Then it's time to not only refactor my classes, but ALSO my tests. Repeat this three or four times and it can get frustrating.

I prefer to write a draft of my classes first then write (and maintain) a battery of unit tests. After I have a draft, TDD works fine for me. For example, if a bug is reported, I will write a test to exploit that bug and then fix the code so the test passes.

Chrass
While you should have an idea of what the architecture of your system will look like, you don't have to know a whole lot ahead of time when doing TDD. TDD means that the tests DRIVE the design, so it will change as you implement more test scenarios
casademora
I agree with the vacuum. The original tutorials of TDD where you'll write the test without ANY code - and get a compile error - is crazy.
mparaz
+3  A: 

Prototyping can be very difficult with TDD - when you're not sure what road you're going to take to a solution, writing the tests up-front can be difficult (other than very broad ones). This can be a pain.

Honestly I don't think that for "core development" for the vast majority of projects there's any real downside, though; it's talked down a lot more than it should be, usually by people who believe their code is good enough that they don't need tests (it never is) and people who just plain can't be bothered to write them.

Calum
A: 

You have to write applications in a different way: one which makes them testable. You'd be surprised how difficult this is at first.

Some people find the concept of thinking about what they're going to write before they write it too hard. Concepts such as mocking can be difficult for some too. TDD in legacy apps can be very difficult if they weren't designed for testing. TDD around frameworks that are not TDD friendly can also be a struggle.

TDD is a skill so junior devs may struggle at first (mainly because they haven't been taught to work this way).

Overall though the cons become solved as people become skilled and you end up abstracting away the 'smelly' code and have a more stable system.

+3  A: 

If your tests are not very thorough you might fall into a false sense of "everything works" just because you tests pass. Theoretically if your tests pass, the code is working; but if we could write code perfectly the first time we wouldn't need tests. The moral here is to make sure to do a sanity check on your own before calling something complete, don't just rely on the tests.

On that note, if your sanity check finds something that is not tested, make sure to go back and write a test for it.

Aaron Lee
+1  A: 

Refocusing on difficult, unforeseen requirements is the constant bane of the programmer. Test-driven development forces you to focus on the already-known, mundane requirements, and limits your development to what has already been imagined.

Think about it, you are likely to end up designing to specific test cases, so you won't get creative and start thinking "it would be cool if the user could do X, Y, and Z". Therefore, when that user starts getting all excited about potential cool requirements X, Y, and Z, your design may be too rigidly focused on already specified test cases, and it will be difficult to adjust.

This, of course, is a double edged sword. If you spend all your time designing for every conceivable, imaginable, X, Y, and Z that a user could ever want, you will inevitably never complete anything. If you do complete something, it will be impossible for anyone (including yourself) to have any idea what you're doing in your code/design.

Doug T.
+32  A: 

If you want to do "real" TDD (read: Test First with Red, Green, Refactor), then you also have to start using Mock/Stubs, when you want to test integration points.

When you start using Mock's then after a while, you will want to start using Dependency Injection (DI) and a Inversion of Control (IoC) container. To do that you need to use Interfaces for every thing (which all has a lot of pitfalls themselves).

At the end of the day, you have to write a lot more code, than if you just do it the "plain old way". Instead of just a customer class, you also to need to write a Interface, a mock class, some IoC configuration ad a few tests.

And remember that test code should also be maintained and cared for. Tests should be as readable as everything else, and it takes time to write good code.

Also many developers don't quite understand how to do all this "right". But because everybody tells them that TDD is the only true way to develop software, they just try the best they can.

But it is much harder than one might think. Often TDD projects ends up with a lot of code that nobody really understands. The Unit Tests often test the wrong thing, the wrong way. And nobody agrees how a good test looks like, not even the so called Gurus agree.

Also all those test, makes it a lot harder to "change" (opposite to refactoring) the behavior of your system, and simple changes just becomes to hard and time consuming.

If you read the TDD literature, they always have some very good samples. But often in real life applications, you have to have a User Interface and a database. This is where TDD gets really hard, and most literature don't have a good answers. And if it does, it always involves more abstractions... mock objects, programming to an interface, MVC/MVP patterns etc., which again requires a lot of knowledge, and... you have to write even more code.

So be careful... if you don't have an enthusiastic team and at least one experienced developer who knows how to write good tests, and also knows a few thing about good architecture, you really have to think twice before going down the TDD road.

Thomas Jespersen
Robert Koritnik
+1  A: 

The downside to TDD is that it is usually tightly associated with 'Agile' methodology, which places no importance on documentation of a system, rather the understanding behind why a test 'should' return one specific value rather than any other resides only in the developer's head.

As soon as the developer leaves or forgets the reason that the test returns one specific value and not some other, you're screwed. TDD is fine IF it is adequately documented and surrounded by human-readable (ie. pointy-haired manager) documentation that can be referred to in 5 years when the world changes and your app needs to as well.

When I speak of documentation, this isn't a blurb in code, this is official writing that exists external to the application, such as use cases and background information that can be referred to by managers, lawyers and the poor sap who has to update your code in 2011.

Ron McMahon
A: 

Having to sack developers who aren't needed any more, because you write working code ;)

Xian
This is a benefit written as a negative, not what the question asker wanted.
ObiWanKenobi
+2  A: 

I've encountered several situations where TDD makes me crazy. To name some:

  • Test case maintainability:

    If you're in a big enterprise, many chances are that you don't have to write the test cases yourself or at least most of them are written by someone else when you enter the company. An application's features changes from time to time and if you don't have a system in place, such as HP Quality Center, to track them, you'll turn crazy in no time.

    This also means that it'll take new team members a fair amount of time to grab what's going on with the test cases. In turn, this can be translated into more money needed.

  • Test automation complexity:

    If you automate some or all of the test cases into machine-runnable test scripts, you will have to make sure these test scripts are in sync with their corresponding manual test cases and in line with the application changes.

    Also, you'll spend time to debug the codes that help you catch bugs. In my opinion, most of these bugs come from the testing team's failure to reflect the application changes in the automation test script. Changes in business logic, GUI and other internal stuff can make your scripts stop running or running unreliably. Sometimes the changes are very subtle and difficult to detect. Once all of my scripts report failure because they based their calculation on information from table 1 while table 1 was now table 2 (because someone swapped the name of the table objects in the application code).

Martin
A: 

Having to explain to upper managment that all those green lines represent progress!

This is a benefit written as a negative, not what the question asker wanted.
ObiWanKenobi
+1  A: 

Let me add that if you apply BDD principles to a TDD project, you can alleviate a few of the major drawbacks listed here (confusion, misunderstandings, etc.). If you're not familiar with BDD, you should read Dan North's introduction. He came up the concept in answer to some of the issues that arose from applying TDD at the workplace. Dan's intro to BDD can be found here.

I only make this suggestion because BDD addresses some of these negatives and acts as a gap-stop. You'll want to consider this when collecting your feedback.

Kilhoffer
Absolutely. You have to consider BDD when evaluating TDD.
I'll second that
It seems BDD = behaviour-driven development
hayalci
A: 

see this article written by me http://muhammadadel.wordpress.com/2007/06/03/test-driven-architecture/ Note article is no longer alive.

Muhammad, for the sake of consistency, post your opinion here, not link
Dragan Panjkov
+3  A: 

On your first TDD project there are two big losses, time and personal freedom

You lose time because:

  • Creating a comprehensive, refactored, maintainable suite of unit and acceptance tests adds major time to the first iteration of the project. This may be time saved in the long run but equally it can be time you don't have to spare.
  • You need to choose and become expert in a core set of tools. A unit testing tool needs to be supplemented by some kind of mocking framework and both need to become part of your automated build system. You also want to pick and generate appropriate metrics.

You lose personal freedom because:

  • TDD is a very disciplined way of writing code that tends to rub raw against those at the top and bottom of the skills scale. Always writing production code in a certain way and subjecting your work to continual peer review may freak out your worst and best developers and even lead to loss of headcount.
  • Most Agile methods that embed TDD require that you talk to the client continually about what you propose to accomplish (in this story/day/whatever) and what the trade offs are. Once again this isn't everyone's cup of tea, both on the developers side of the fence and the clients.

Hope this helps

Garth Gilmour
+2  A: 

The biggest downside is that if you really want to do TDD properly you will have to fail a lot before you succeed. Given how many software companies work (dollar per KLOC) you will eventually get fired. Even if your code is faster, cleaner, easier to maintain, and has less bugs.

If you are working in a company that pays you by the KLOCs (or requirements implemented -- even if not tested) stay away from TDD (or code reviews, or pair programming, or Continuous Integration, etc. etc. etc.).

Vasco Duarte
A: 

It takes some time to get into it and some time to start doing it in a project but... I always regret not doing a Test Driven approach when I find silly bugs that an automated test could have found very fast. In addition, TDD improves code quality.

+1  A: 
  • unit test are more code to write, thus a higher upfront cost of development
  • it is more code to maintain
  • additional learning required
Bob Dizzle
+22  A: 

Several downsides (and I'm not claiming there are no benefits - especially when writing the foundation of a project - it'd save a lot of time at the end):

  • Big time investment. For the simple case you lose about 20% of the actual implementation, but for complicated cases you lose much more.
  • Additional Complexity. For complex cases your test cases are harder to calculate, I'd suggest in cases like that to try and use automatic reference code that will run in parallel in the debug version / test run, instead of the unit test of simplest cases.
  • Design Impacts. Sometimes the design is not clear at the start and evolves as you go along - this will force you to redo your test which will generate a big time lose. I would suggest postponing unit tests in this case until you have some grasp of the design in mind.
  • Continuous Tweaking. For data structures and black box algorithms unit tests would be perfect, but for algorithms that tend to be changed, tweaked or fine tuned, this can cause a big time investment that one might claim is not justified. So use it when you think it actually fits the system and don't force the design to fit to TDD.
Adi
A lot of "graphic effects" will be based on geometry which can easily be applied to testing. Otherwise it is based on a library that should be well tested
Tom Martin
Graphic Effects or UI elements?
IanL
The main point (4) is - any system which is not well defined and is likely to keep changing to match an evolving visual behavior, different AI spec, behavioral algorithms, etc.. will cause big time investment in repeated test definitions since we keep on changing the desired test results.
Adi
@Adi: True, but wouldn't that be the same without TDD? Without it, you'd have to do more manual testing, which would suffer the same problem.
sleske
Isn't the "Big time investment" going to save you time later on while developing your solution? Especially with a complex one? I guess it should save you time. Not to think about maintenance phase where small changes can easily break the system. ( **or maybe I'm just being naïve about unit + regression tests preventing future bugs** )
Robert Koritnik
A: 

You have to make sure your tests are always up to date, the moment you start ignoring red lights is the moment the tests become meaningless.

You also have to make sure the tests are comprehensive, or the moment a big bug appears, the stuffy management type you finally convinced to let you spend time writing more code will complain.

qui
+2  A: 

The biggest problem are the people who don't know how to write proper unit tests. They write tests that depend on each other (and they work great running with Ant, but then all of sudden fail when I run them from Eclipse, just because they run in different order). They write tests that don't test anything in particular - they just debug the code, check the result, and change it into test, calling it "test1". They widen the scope of classes and methods, just because it will be easier to write unit tests for them. The code of unit tests is terrible, with all the classical programming problems (heavy coupling, methods that are 500 lines long, hard-coded values, code duplication) and is a hell to maintain. For some strange reason people treat unit tests as something inferior to the "real" code, and they don't care about their quality at all. :-(

Skillwired
A: 

The person who taught my team agile development didn't believe in planning, you only wrote as much for the tiniest requirement.

His motto was refactor, refactor, refactor. I came to understand that refactor meant 'not planning ahead'.

Jack B Nimble
+2  A: 

In the few years that I've been practicing Test Driven Development, I'd have to say the biggest downsides are:

Selling it to management

TDD is best done in pairs. For one, it's tough to resist the urge to just write the implementation when you KNOW how to write an if/else statement. But a pair will keep you on task because you keep him on task. Sadly, many companies/managers don't think that this is a good use of resources. Why pay for two people to write one feature, when I have two features that need to be done at the same time?

Selling it to other developers

Some people just don't have the patience for writing unit tests. Some are very proud of their work. Or, some just like seeing convoluted methods/functions bleed off the end of the screen. TDD isn't for everyone, but I really wish it were. It would make maintaining stuff so much easier for those poor souls who inherit code.

Maintaining the test code along with your production code

Ideally, your tests will only break when you make a bad code decision. That is, you thought the system worked one way, and it turns out it didn't. By breaking a test, or a (small) set of tests, this is actually good news. You know exactly how your new code will affect the system. However, if your tests are poorly written, tightly coupled or, worse yet, generated (cough VS Test), then main

Writing tests so that you cover everything (100% code coverage)

Ideally, again, if you adhere to the methodology, your code will be 100% tested by default. Typically, thought, I end up with code coverage upwards of 90%. This usually happens when I have some template style architecture, and the base is tested, and I try to cut corners and not test the template customizations. Also, I have found that when I encounter a new barrier I hadn't previously encountered, I have a learning curve in testing it. I will admit to writing some lines of code the old skool way, but I really like to have that 100%. (I guess I was an over achiever in school, er skool).

However, with that I'd say that the benefits of TDD far outweigh the negatives for the simple idea that if you can achieve a good set of tests that cover your application but aren't so fragile that one change breaks them all, you will be able to keep adding new features on day 300 of your project as you did on day 1. This doesn't happen with all those who try TDD thinking it's a magic bullet to all their bug-ridden code, and so they think it can't work, period.

Personally I have found that with TDD, I write simpler code, I spend less time debating if a particular code solution will work or not, and that I have no fear to change any line of code that doesn't meet the criteria set forth by the team.

TDD is a tough discipline to master, and I've been at it for a few years, and I still learn new testing techniques all the time. It is a huge time investment up front, but, over the long term, your sustainability will be much greater than if you had no automated unit tests. Now, if only my bosses could figure this out.

casademora
What was the rest of the sentence ending with "(cough VS Test), then main"?
Andrew Grimm
A: 

Good answers all. I would add a few ways to avoid the dark side of TDD:

  • I've written apps to do their own randomized self-test. The problem with writing specific tests is even if you write lots of them they only cover the cases you think of. Random-test generators find problems you didn't think of.

  • The whole concept of lots of unit tests implies that you have components that can get into invalid states, like complex data structures. If you stay away from complex data structures there's a lot less to test.

  • To the extent your application allows it, be shy of design that relies on the proper ordering of notifications, events and side-effects. Those can easily get dropped or scrambled so they need a lot of testing.

Mike Dunlavey
+2  A: 

You lose the ability to say you are "done" before testing all your code.

You lose the capability to write hundreds or thousands of lines of code before running it.

You lose the opportunity to learn through debugging.

You lose the flexibility to ship code that you aren't sure of.

You lose the freedom to tightly couple your modules.

You lose option to skip writing low level design documentation.

You lose the stability that comes with code that everyone is afraid to change.

You lose the title of "hacker".

Uncle Bob
+1 This is the perfect answer. Thanks, Uncle Bob.
Torbjørn
You also lose the ability to deliver a solution on time.
MagicAndi
Depends on your definition of "deliver a solution on time" - is that "any old, partially broken solution on time" or "deliver working solutions on time". You certainly loose the ability to deliver partially broken solutions on time. As far as dev speed goes - I like the metric "elapsed time between start of dev and a week of faultless live deployment". If you measure it fairly, it's hard to even stop the clock at all on a copmlex piece of non-TDD work.
cartoonfox
-1, this is exactly the thing the OP said he didn't want.
erikkallen
Many true statements, but: what erikkallen said. -1.
j_random_hacker
@ j_random_hacker says hacker... LOL
Dan
A: 

Development time increases : Every method needs testing, and if you have a large application with dependencies you need to prepare and clean your data for tests.

Mona