What can't be solved by TDD?

Question

I was just perusing the TDD tag and noticed that the 1,000th TDD question was due to be asked. I figured it should be a big one - and I guess that means community wiki.

TDD solves many things for us - guarantees testable code, promotes low coupling and high cohesion, enhances readability and generates its own "executable documentation". But what programming problems does it not solve? What problems have you run into that TDD can't touch?

For extra credit: what techniques have helped you solve those problems that TDD couldn't touch?

Answer 1

TDD can't solve bad design or architecture issues or poor algorithm choices or just plain old stupidity.

TDD can't make quality free - there's a cost associated with it that zealots tend to dismiss.

Answer 2

TDD can't prevent or cure stupidity.

(I am basing this on the amount of silly and useless unit tests I have seen).

Answer 3

TDD is not silver bullet. It's a way out of many. It cannot replace functional and intergation tests.

Answer 4

There are a number of things it can't solve:

1. Can't address poor or incomplete requirements.
2. Can't work as a silver bullet for unrealistic project deadlines.
3. Can't fix poor architecture choices.
4. Can't guarantee a robust development environment, it's only a technique.
5. Won't solve writers/coders block. You'll get stuck on what test to write next.
6. Doesn't fix a poor understanding of the problem domain. It will, however, show that the person doesn't know what to test to someone who is knowledgeable.
7. Won't show you the answer to what algorithm is the "best" to use.
8. Isn't a guarantee that you can push to production from development. It has to go to test unless you're willing to accept the possibility of bugs.

That's all I can think of. About the only way around the things that TDD doesn't address is to have a competent, team-oriented group who want to make good software rather than clocking in/out from 9 to 5.

Answer 5

TDD only checks that the code is doing what is was written to do. Its not verifying the code is actually solving the problem that it was written to complete.

But depending on experience with TDD will vary on what you can do with it. I've only scratched the surface of it for my final year project and there was plenty I could not achieve with it both on inexperience and lack of time to use it to its fullest extent.

I'm having a vague recollection that on my work placement we had issues with TTD when using reflection to create new classes but I am sure we did cure any issues with it.

But +1 to the comments above, TTD verifies the code given enough test cases, but not the design or requirements.

it maybe worth mentioning at this point I wrote ~15 test cases for a segment of code I wrote. And the first time someone else used my code in the project it broke. The one usage I did not account for was the one they used, and the code did not cope with that set of input ;) So it does go to show its not a silver bullet, more of a confidence check.

Answer 6

TDD cannot help you to solve a problem if you don't know how to solve it. In the book "Coders at Work" by Peter Seibel there was a great example in his interview with Peter Norvig. He talked about a Sudoku solver that he wrote that some big TDD blogger also attempted to write. The TDD guy writes a bunch of tests in about 5 blog posts and then that's it, he didn't know how to solve the problem at all. Peter Norvig knew how to solve the problem because he knew constraint propagation and recursive search and so he could see how to solve the problem. Anyway I have the Kindle Version of the book so it says p. 3773-3787, not sure what that translates to in the physical book. Maybe 377-378.

But the point is if you don't know how to solve the problem, you can write all the tests in the world, they won't magically give you the solution. Anyway Norvig has the quote in that book "you can test all you want and if you don't know how to approach the problem, you're not going to get a solution."

Aside from that a lot of other people made many great points. In particular tests don't fix stupidity, poor efficiency, bad design, etc...

Another point is that TDD is no substitute for QA/system tests. The author of the code is always suspect when testing his/her own code. There will always be cases the author thinks are so simple they must be right, or assumptions built in that will result in not testing things (oh no one will ever use this that way...). The job of QA is to think of things that the author never would. I remember some medical device where people somehow used the arrow keys to arrow back over a number and then enter a new radiation dose thinking they were deleting the old numbers, but they were just adding more numbers and the device was over exposing patients. The author did not think of that case. Those are the things that a good QA department with experts in breaking stuff might be more likely to find. Basically TDD is great for integrating unit tests (and maybe even some smaller integration tests) with the development process. But someone needs to think of tests from the point of view of an outsider and test everything together.

Some systems have "emergent" properties. Basically properties that only show up when you put all these different components together. Those need to be tested as well. Maybe you put together 10 modules and each stores 100 MB of data. So you have a 500 MB hard disk and think you are safe for each module but then when everything is together it takes 1 GB. Basically stuff like that will always need some type of system integration tests because problems always arise as a consequence of the interactions and no one is going to see them all.

Answer 7

One point I find important: one can create a zero bugs application with TDD that doesn't sell at all. TDD doesn't ensure that you're building something useful.

Answer 8

I think there's way too much idealism around TDD.

TDD is not magic. It's a methodology to help REDUCE bugs or errors. Many people seem to assume it magically makes programming bulletproof.

What are bugs? Mistakes, made by the programmer (or maybe a requirements person, etc).

How does TDD help? It insists that you write as many tests as you can think of, and that you write the tests first... so that every bit of code has some sort of test covering it.

However:

• Programmer fallibility is one of the main reasons for TDD, and so obviously you're going to get the tests themselves wrong just as often.
• All you can say about code written to satisfy a set of tests is that it passes those tests. It doesn't say that the tests are right, or the code properly handles situations you haven't thought of...
• Claiming that the tests define the requirements and an incomplete test coverage situation means the requirements were wrong (not the code) is just a semantic technicality... it's still wrong and you're just playing a blame game.
• You're obviously writing a lot more code, and now more needs to change as requirements change.
• TDD is good at the component level, but at the systems level it gets fuzzy. As systems get more complex and the number of possible interactions go up, you're going to be writing a huge amount of crazy test code, and it's going to be harder and harder to instantiate and prepare everything you need in each test. Properly segmenting module responsibilities and tasks will help here, but at some point it will simply become too complex.
• Mocking external systems can only get you so far. If an external system has a problem, very often you have no recourse but to work around the bug. Mocking based on the ideal external component isn't going to guarantee anything at all. The tests are going to turn up SOME issues in YOUR code, but not all issues by any stretch of the imagination.
• TDD is a methodology that works best on projects that are highly specified up front. There are many problem domains where a highly iterative development process works much better, and iterative problems are best solved with a well thought out architecture designed for handling change. TDD processes at best are entirely orthogonal to that, and at worst add a substantial burden.

My personal opinion is that writing tests first, and only writing code to satisfy the tests and no further... is a concept that MOST people get horribly wrong. In the hands of novices it promotes an unthinking, reactionary code design... you're coding for tests, not to any sort of architectural vision.

Programming is about problem solving and design, not about learning how to apply a process.

The example mentioned by Cervo of a "big TDD blogger" that couldn't solve Sudoku is just totally demonstrating the problem. People THINK they're becoming highly skilled and effective, but they're just getting better at a process... not the core skills of their profession. I would not take art lessons from someone who could only draw stick figures, no matter how perfect he claimed his process was.

I personally believe TDD has value in providing validation that your components or API does not break in known ways between changes... but that's about it.