How to avoid the 80/20 rule in software development

Question

It seems that no matter what my project is, I get through 80% of the work fairly fast. Users and management get excited thinking things are way ahead of schedule, but the pesky 20% of work remaining seems to take 4 times as long as the previous 80%. When we have our regular check ins or stand ups on the project, I feel like a broken record saying "yes things have gone OK so far, but there is still quite a bit left to do..."

For the most part, my estimates are fairly accurate, but I am human. What is the best approach for convincing users that the last 20% of work really does take 80% of the time? It seems like more and more users and management believe IT is easy and magic happens at the snap of some fingers...

In general, we do track tasks at what I believe to be a fairly low level. Not necessarily at a create label or textbox, but we are pretty detailed... We also track our estimate to completion on all tasks, which I feel is a more important number than the original estimate when you're in the middle of the project.

I think it comes down to the perception of the users and management. Even though they may know the estimate to completion, they still get wrapped up in the emotions and perceptions on what they are seeing and the estimate numbers take a back seat. This is what I'm trying to figure out how to contain or manage expectations to.

---

EDIT
Turning into a community wiki as this is rather subjective. Should have been that way from the beginning.

Answer 1

It always takes longer than you expect, even when you take Hofstadter's Law into account

But i digress.

The best practice is sadly experience. SCRUM methods have been very helpful for some types of software development since they constantly update the release date to a more accurate date. (quick video about scrum)

Answer 2

I have found that the best way is to keep the schedule up-to-date with percentage complete for each task. That way, the progress is quite clear. Communicate that up to management and they should understand where you are at all times.

Answer 3

Perhaps you need to break down the tasks/functionality you are working on into smaller units, both for scheduling work and for checkins/reports. For example I almost never have any individual item on my schedule that lasts longer than two days.

Then instead of saying "I'm working on our new muppet maker" each day at Scrum for two weeks you can then say "I'm currently working on the eye selector for the muppet maker".

If you are working from schedules, and your schedules are accurate (meaning they account for both the 80% & the 20%), then management really shouldn't have a problem. If they imply they can reduce the allotted time because you are "almost finished" then show them the parts of the spec that have not been implemented.

I'm assuming that you work from some form of functional specification that details what something should do, how it should behave, and the edge cases it must deal with. If this is the case then worrying about the emotions and perceptions of management seems very strange to me, they should be quite capable of either comparing the spec to your work, or reading your schedule to see what is left.

Answer 4

I think that if you're making estimates for a task that bundle together the 80% and the 20%, you're off to a weak start. Break apart your estimates. Make the 80% and the 20% two explicit tasks; the easy bit and the hard bit, if you must.

Then you can provide more realistic time estimates up front for the work as a whole, and make it easier for production to track the specifics.

Answer 5

Don't show them the first 80% as soon as it is finished. Drip feed them.

Answer 6

When planning, take it into account. When estimating the time it'll take to do a subtask, give an estimate for "Done" rather than "Basically done" (Estimate from the experience how much longer "Done" takes. Resist the temptation to dismiss integration, documentation, tidying up, testing, bug-fixing after testing, deployment etc as small tasks that will be absorbed into other tasks.

You'll end up with a horrifying large estimate. But ask yourself whether it's out of line with the actual time spent on previous projects.

Answer 7

Try and provide the most accurate estimates you can and provide as much transparency into the project as possible. If you are consistently close on your estimates that should be enough to satisfy your managers. Remember it is very very difficult to measure productivity.

Answer 8

How do you estimate the amount of work? You say that "the pesky 20% of work remaining seems to take 4 times as long as the previous 80%", but how did you get to the estimate that "20%" of the work is remaining and that "80%" is done? Obviously the estimates are wrong - in reality only 20% of the work is done and 80% is remaining.

In software development it's very hard to give accurate estimates long time in advance. The only way is to split the work into small manageable pieces (maybe less than 10 hours each). You can estimate accurately only the immediate next steps.

Some practices which help in estimating progress can be found in Scrum. The scope of what work will be done during the next sprint (one month or less) is fixed at the beginning of the sprint and rough estimates are given to each work. Then after the sprint the team can reflect on how much progress was done, how much is still missing, how accurate the estimates were, and what is slowing down the team. In Scrum and other agile methods an important point is getting fast feedback of what is done and how far we are in the project. I recommend reading more about them. The video about Scrum that Ólafur Waage linked in his message gives a good and quick introduction.

Answer 9

It has been said that the first 90% of the project time is used for 90% of the work, and the remaining 90% of the project time is used for the remaining 10% or the work. ;)

It's natural to make great progress at the start of the project, as you simply do the easiest parts first. Also, if there are any problems in the first 80% of the code, they will often not be evident until it all comes together and you can actually test all of the code.

Perhaps as an experient you should let people try the application that is 90% done so that they see what a difference the last 10% does...

Answer 10

I think it comes down to the perception of the users and management. Even though they may know the estimate to completion, they still get wrapped up in the emotions and perceptions on what they are seeing and the estimate numbers take a back seat. This is what I'm trying to figure out how to contain or manage expectations to.

Write the algorithms first... then the UI.

Answer 11

I've discovered a few things that greatly help with time estimates

1. Familiarity with the codebase. When you can listen to the spec and can think "I need to touch class A, B, and C - nothing more, nothing less", then you can get pretty accurate. I find this works better than knowing which specific functions need to be written because you then know what you don't need to write.
2. Remembering to include testing, bug fixing, deployment, and last-minute requests. Its easy to forget that you need to migrate a bunch of records.
3. To a certain extent, being familiar with the language. If you know which libraries you'll need, then it becomes easier to know what you don't have to do.

I've used this pretty successfully in approximating coworker's speed, too, It just takes some empirical observations about how fast they can develop a feature and how good it'll be prior to actually testing it.

Answer 12

Read Steve McConnell's excellent book Rapid Development which has much to say regarding the 80/20 issue and the othder vagaries of software estimation.

Answer 13

Keep your time horizons short. It is easier to estimate what you will be doing in the next several weeks than the next several months. Consider breaking your project into milestones that are short. A month or perhaps a couple months depends on what you are trying to do. This is basically what Scrum does. Then estimate most accurately only the work you are going to do in the current milestone. Re-estimate the next milestone when you get there and have a lot more data to base the estimates on. Part of the reason the last 20% is taking so long is that you likely are making the estimate up front when you don't know enough.

Also, try Wideband Delphi estimation techniques. This will tease out a lot more of the hidden costs that you likely aren't considering.

Answer 14

When it comes to time estimating this is my experience:

1. If you can't positively say that a task will take less than 4 hours you can't estimate it accurately. Break it down in smaller pieces and repeat recursively.

2. Making such a time estimate is no picnic, it will take time, you will basically have to iron out the complete project in manageable chunks meaning that any changes to the requirement will result in a changed time-plan (surprising, isn't it?)

3. The biggest problem is that we can't possibly foresee all the details (maybe, let's say 20% perhaps? Leaving you the rest 80% unestimated...) - see SCRUM as others already have pointed out.

4. Management will seldom "accept" such a detailed time estimate as it will "take too long" to implement.

However, as management is interested in making profit, they are also interested in cutting corners. So you should identify the corners possible to cut and make sophisticated compromises based on the real life scenarios involved. Backed by management you can accomplish a lot of these last 20% by doing nothing (sad in a way I guess, but still true).

Because the last 80% of the work which represents the last 20% of the final product is really polishing and ironing out bugs and adapting to changed requirements, etc. It might be possible to have some limited first version, etc, etc, be creative.

Answer 15

Saying "yes things have gone OK so far, but there is still quite a bit left to do..." might not be the best way to get your point across. After hearing that a manager or client might think "yes, there is quite a bit left to do but he did this part so fast surely the rest is minimal".

Instead make sure to identify the remaining work and schedule them. This way you can show where you should be during th remaining 20% of the project. If it's taking too long your schedule will show a project behind schedule and that should raise some sense of urgency.

Keep your tasks updated weekly (or how ever regularly you have status reports). Identify areas that are in danger of falling behind especially if other areas depend on them.

Answer 16

I don't think I can say it better than Joel does with Painless Software Schedules.

If your manager makes you reduce an estimate, here's what to do. Create a new column in the schedule called Rick's Estimate (assuming your name is Rick, of course.) Put your estimate in there. Let your manager do whatever she wants with the Curr Est column. Ignore your manager's estimates. When the project is done, see who was closer to reality. I've found that just threatening to do this works wonders, especially when your manager realizes that they've just gotten into a contest to see how slowly you can work!

Answer 17

One of the root causes of the 80/20 phenomenon is that the unexpected always occurs for any difficult - and sometimes even trivial - tasks. For example: the documentation that your software design processes mandate suddenly get a new template format, thanks to some overzealous process managers. Suddenly, it isn't just a simple matter of updating the docs for your new release - you now have to restructure each of them, and all of them take significantly more time.

One of the best recommendations I've heard for handling this type of phenomenon is to always build buffer subtask(s) into the project schedule - recommended by Richard Whitehead. Every major task gets a 20% time increase (or somewhere thereabouts) noted as a subtask for that task. The purpose of each is to provide some measurement for what happens when "things go wrong" on that task. The author admits (and I've also found to be true) that often management will try to remove those buffer tasks - your only recourse is to either stand your ground, or pull a maneuver like Joel advocates (as @Casey already mentioned). In practice, I've found that a good number of buffer subtasks usually do stick around, and have helped out a few times in tight schedules.

Answer 18

I think it is best if you can to under-promise and over-deliver.

If your estimates are right, then you account for that pesky 20%. You obviously did not, and that is why it is an issue.

Maybe you are trying to give them everything they want which is unrealistic. Maybe you did not fully account for Murphy's Law, or did not give enough time for testing, finding bugs, then testing again, etc.

Looks like you should be doing a little more risk management...

Answer 19

If you're consistently finding that the last 20% of work is taking 4x the time of the first 80%, then it may be time for an honest discussion with yourself (or with trusted peers) about whether you're letting technical debt build up during the first 80%, and it's biting you at the end.

That may speak to work practices that you could consider changing.

Two of the best practices I know of for keeping technical debt down are writing good tests (preferably before you write the code, but just afterwords works, too), and refactoring after every tasks. Think of refactoring as cleaning the kitchen after every meal, instead of at the end of the month.