How do you avoid Technical Debt while still keep true to Agile, i.e.: avoiding violation of YAGNI and avoiding BDUF?

Question

Technical Debt via Martin Fowler, via Steve McConnell

YAGNI (You Ain't Gonna Need It) via Wikipedia

BDUF (Big Design Up Front) via Wikipedia

UPDATE: To clarify the question, I think I can also state it this way and keep my meaning:

"In what ways do you, as an Agile practioner, find the right balance between "quick and dirty" (unintentionally risking Technical Debt while attempting to adhere to YAGNI) and over-engineering (BDUF) within each iteration?"

Answer 1

I find Robert Martin's Test Driven Development (TDD) approach helps with these concerns.

Why?

• You only have to write enough code to pass the next test.
• I think testable code is cleaner.
• The design has to feed into tests which can help keep the design focused.
• When you do have to change (refactor) you have tests to fall back on

Regardless of when the test are written (before or after) I find writing the test helps you make practical decisions. E.g., we picked design A or B because A is more testable.

Answer 2

That's why it's always easier to write nice "acadamic papers" talking about how Agile development is good, what are the "best practices" and so on.

That's why you find a lot of "suited engineers" making up new software engineering techniques.

Process is important, keeping best practices is cool but over any other thing, common sense drive design process. Software is developed by people so YAGNI really should be:

I might not gonna needed but maybe I will because in my concrete bussiness/company/department this thing do happen or I will need it but I just haven't the time right no so quick and dirty hack to make the cash and keep my job, or I might need it and refactoring later will be a pain in the ass costing 10 times more than just doing it now from the scratch, and I have the time NOW.

So use your common sense, trust it or trust the common sense of the people working for you. Don't take every academic paper as proven fact, experience is the best teacher and your company should improve their way or making things with time and its own experience.

Edit: Incidentally, TDD is the opposite of YAGNI you're building test before even knowing if you are gonna need them. Seriously, stop listening to academics!! There's no magical way to produce software.

Answer 3

The 'traditional' XP answer is refactoring combined with automated unit testing.

But it's a very interesting question philosophically. I don't believe you need to avoid technical debt, just keep it at a manageable level. Steve McConnell's article is good on this balance - the reason the analogy works is that it's normal and acceptable to build up financial debt in a company, as long as you accept the costs and risks - and technical debt is fine too.

Maybe the answer itself also lies in the principle of YAGNI. You Ain't Gonna Need the technical debt paid off until you do, and that's when you do the refactor. When you're doing substantial work on an area of the system with technical debt, take a look at how much short-term difference it will make to do the redesign. If it's enough to make it worthwhile, pay it off. McConnell's suggestion of maintaining a debt list will help you to know when to make this consideration.

I don't suppose there is an absolute answer to this - like many things it's a judgment call based on your experience, intuition and your analysis in each particular situation.

Answer 4

Surely being agile is going to keep your TD down for any given project?

The fact that you are implementing what the customer wants, i.e. with their feedback, is keeping TD to a minimum,

Answer 5

There was an interesting discussion of Technical Debt based on your definition of done on HanselMinutes a couple of weeks ago -- What is Done. The basics of the show were that if you re-define 'Done' to increase perceived velocity, then you will amass Technical Debt. The corollary of this is that if you do not have a proper definition of 'Done' then you most likely are acquiring a list of items that will need to be finished before release irrespective of the design methodology.

Answer 6

It seems that if you stick with the "plan, do, adapt; plan, do, adapt" idea of agile (iterations, iteration reviews) you would avoid those things by default. BDUF is just so contrary to the idea of agile estimating & planning that if you are really agile, you wont be BDUF automatically.

The purpose of release & iteration planning meetings is to make sure you are adding the most valuable features to the project for that iteration. If you keep that in mind, you'll avoid YAGNI for free.

I would very strongly recommend the Mike Cohn books on agile planning:

1. User Stories Applied
2. Agile Estimating and Planning

---

Update: After your clarification about avoiding YAGNI and BDUF within an iteration...

BDUF...If I felt a feature was not clearly defined before I started work on it, I would create a small "feature" or story to account for the design type portion of the work needed. So that maybe the smaller story has a story point estimate of 1 instead of the real feature's 5. That way, the design is time-boxed into the smaller story, and you will be driven to move on to the feature itself.

To avoid violating YAGNI I would work to be very clear about what the customer expects for a feature within an iteration. Only do work that maps to what the customer expects. If you think something extra should be added, create a new feature for it, and add it to the backlog of work to be done. You would then persuade the customer to see the benefit of it; just as the customer would push for a feature being done at a certain point in time.

Answer 7

You seem to say that "YAGNI" implies "quick and dirty". I do not see that.

As an agile programmer, I practice test-driven development, code review and continuous integration.

• Test-driven development (TDD), as a process, is a good way to avoid YAGNI. Code that's just there "in case it will be useful" tends to be untested and hard to test.
• TDD also largely removes the compulsion to BDUF: when your process is to start by sitting down and start doing something that actually delivers value, you cannot indulge in BDUF.
• TDD, as a design practice means that the big design will emerge as you gain experience with the problem, and refactor real code.
• Continous integration means that you design your process so your product is essentially releasable at any time. That means that you have a integrated quality process that tries to prevent the quality of the mainline from dropping.

In my experience, the main forms of technical debt are:

• Code not covered by the automated test suite. Do not allow that to happen, except for very localized components that are especially hard to test. Untested code is broken code.
• Ugly code that violates the coding standard. Do not allow that to happen. That is one of the reasons why you need to build code review into the continous integration process.
• Code and tests that smell and need refactoring to be more easily modified or understood. This is the benign form of technical debt. Use your experience to know when to accumulate it, and when to repay it.

Not sure if that answered your question, but I had fun writing it.

Troy DeMonbreun commented:

No, that wasn't my point... "quick and dirty" = (unintentionally risking Technical Debt while attempting to adhere to YAGNI"). That does not mean YAGNI is only quick and dirty. The phrase "quick and dirty" is what I used to quote Martin Fowler in his description of Technical Debt

Avoiding YAGNI is another way of saying KISS. YAGNI increases the technical debt. There is no tension between between avoiding YAGNI and keeping the technical debt low.

I think I might still be missing the point of your question.

Answer 8

Just do the simplest thing that works. I agree with Ayende when he says that the key to being agile is to "ship it, often". A regular release cycle like this will mean that there is no time for BDUF and it will also dissuade developers from violating YAGNI.

Answer 9

Where I work, I believe the way we avoid the debt is by spinning through the cycle quickly, i.e. showing functionality to the would be end user and get either a sign off that it should be pushed to test or a rejection to say what was wrong giving an updated requirement. By doing this repeatedly within an iteration, a lot of changes can be found about what the user wants by trying this and that.

A key point is to try to do exactly what the user wants as doing more violates YAGNI and brings in BDUF while the idea of refining the requirements over and over again is at the heart of Agile to my mind.