Why are initial estimates taken for granted?

Question

I find it kinda funny that if you are asked how long you would estimate to perform a specific task, your answer is always the amount of time you have to finish the task – maybe with 1-2 additional days.

Does anyone else experience this? Furthermore, why is the pressure always on the developer? Yes, he gave a false estimation, which can happen (...often). Surely your estimations get more accurate the more often you do them and the better your knowledge about your project is...

But I wonder why – because an estimation is always an estimation – one cannot predict the future, but from a project manager's point of view a developer certainly can... or should. :)

Answer 1

Developers are just notoriously bad at estimating how long something will take.

It does not always make sense to estimate in days. I have used things such as story-points, t-shirt sizes, tomatoes, etc. Find out what these abstract measures translate to in hours/days and your estimates will get better.

Answer 2

The problem is that the people who are asking for estimates don't understand our job. They don't understand that so many unexpected problems can crop up, and how even the most experienced developer can have an oversight. Things that seem trivial can turn out to be complex and solutions that seem solid can fall apart half way through, both of which lead to longer time.

As a rule of thumb, it's wise to give your estimate as (how long you think it takes * 2).

Answer 3

You asked a why question...

... to which the answer is always "people are stupid".

This is something I struggle with every day. We don't do the same thing over & over... we do something novel almost every time we sit down at the keyboard -- so estimation is always a guess.

Now, there are development methodologies that try to manage this unknown (take a look at some of the agile practices), but those require management buy-in which is really hard to get.

Answer 4

The estimation you provide is considered a deadline. In most cases, instead of forcing a deadline on you, a good manager will ask you when you can deliver and hold you to it. You need to remember that most successful businesses focus on deliverables and timeframes, not "hope" (as in "I hope to be done soon").

Answer 5

Yes i have experienced the same thing. Probably most of us have. And i tend to give optimistic estimates. The trick is to multiply them by pi (or some other factor), so you get a fairly realistic estimate.

Answer 6

You might find this very relevant (if not essentially a duplicate):

http://stackoverflow.com/questions/208477/need-advice-on-how-to-give-accurate-estimations-on-project-completion/

Answer 7

From what I have heard when estimating a project. Make a good estimation and then double it. This will normally be the amount of time it will take to complete the project

Answer 8

Estimates are always taken for granted because people want to believe in rationality. So it does not matter if your manager learns to multiply all you say by 3. She just needs something seemingly rational to work with.

That being said, estimates worked out by a group of developers seem to give good results.

Answer 9

Read the book "Waltzing With Bears" for a good explanation and introduction of a very important tool: the probability graph of task completion time vs estimates, and using two-point estimates as a counterstrategy.

Answer 10

It is an observable fact, that even the estimates loudly labelled WAG are taken as gospel, engraved on tablets of stone, never to be changed. This is largely because of the usual set of human foibles, including laziness and incomprehension on part of those requesting the estimate. Consequently, people get into the habit of ensuring that their estimates are safe - they are able to meet the time line of their estimate. This is also known as 'padding', but it is a basic self-protection. If your initial guess is going to taken as the final result, and missing your initial guess will cause angst later, you make sure you are not afflicted with angst by making sure your guess is achievable.

Answer 11

What I find more amusing is the way estimates become quotes when looked at retrospectively. For example, the following scenario.

Project Manager: Can I get an estimate of how long this will take?

Programmer: My estimate is 3 days.

<3 days later>

Project Manager: Why isn't it done yet? You promised me it would take 3 days.

Answer 12

Management does not understand what programming takes unless they are experienced programmers still coding with you.

When managers try to expect a certain deadline, it is almost entirely business driven. When it is not, they draw false conclusions based on past performance, thinking it will be easy because it's almost the same as your last work. But that "almost" sometimes costs a lot more time.

Answer 13

While many project managers don't understand what developers do it's equally true that many developers don't understand what project managers do. Time lines are important to the business (you remember the business...the folks who make the money that pays your rent). If you work for a software company, the sales folks need to be able to tell their customers when the new product or feature will be available to them. If you work in an internal development organization, the users of the software need to know when the software is going to be ready so they can plan process changes, training and the like.

Now that I've said that, estimating is REALLY HARD for developers. But, it's something we can get better at. When you make an estimate and it turns out to be wrong, look at why it was wrong. What did you miss when you were looking forward in time that you can now see looking backward. Next time you make an estimate use this new knowledge to make a better estimate. Was the some complexity you missed? Did you need to implement some complex new logic? Why did you miss the mark? Like everything we do we should work at getting better at estimating.

Answer 14

Estimates are "taken for granted" because the person asking you for the estimate often has no other choice than to believe you. After all, you are the expert, not they, how can they second-guess your estimate?

In my organisation estimates are often challenged like this: the estimator is asked for a detailed breakdown of how he or she arrived at the value. We also work with two-point estimates - a 75% certainty estimate and a 95% certainty estimate.

This helps keeping things in line, correct, and realistic.

Answer 15

You can't possibly know, up front, everything you will do, nor could you derive it before actually starting the work. Concrete numbers are impossible.

But someone has to clear you to do that work, decisions must be made, inertia must be started, and those decisions require input: concrete numbers are implied.

This leads to a form of Cognitive Dissonance, which expresses itself as strife between those involved (as it always does).

The temptation to ignore the fact that the problem changes while you work on it also leads to the same dissonance.

Answer 16

Programming is never the same thing twice. Sure, major concepts may be the same, but if a program is going to be exactly the same as another program, then why bother writing it in the first place? Since a new program is always just that - new - there is no way to accurately predict the time it will take to invent it.

As an analogy, suppose a company is contracted to design the next fighter plane - the (fictitious) "F-1234". Sure, it's a plane. Sure, major components are the same. It has engines and wings and weapons and landing gear. Sure, it's very similar to hundreds of other planes. However, whatever the contract granter is looking for in this plane to make it unique needs to be invented. No one can predict the time it will take to invent something, else it would have needed to already have been done to know how long. This is why projects for new fighter planes (and software!) are almost never in line with a so-called project management time schedule.

Answer 17

If asked for an estimate of when a task will be done then sometimes this will work:

You can somewhat estimate the work of doing the task at hand.

You then need to add time for parts of the task you did not think off. And time spend debugging your work.

Then you need to add time for the time you do not actually work on that task. Higher priority tasks show up, vacation, kids get sick, meetings, existing work queue.

So sometimes something like this works:

Estimate time needed for task as you envision it
Based on complexity of task multiply by 1 to 8
Based on relative priority of task multiple by 2 to 8

Then round it up to the next day

Answer 18

Estimates are inaccurate when any of these three things apply:

1. You don't understand the problem that you're being asked to solve.
2. The end user doesn't understand the problem that you're being asked to solve.
3. The solution will require you to do something you haven't done before.

The thing is, these three things almost always apply. This means that estimates are almost always inaccurate.

If you look back at inaccurate estimates (something that is usually only done during blamestorming sessions), it's usually fairly easy to see where things went wrong. "When the users said that we didn't want to process any cases for which this condition was true, we didn't realize that the condition couldn't be determined from the data in the database." "It took me a long time to get all of the IBindingList<T> members implemented properly." "We had to spend a lot more work on the UI than we thought we would."

The fallacy is thinking that you, or some smarter, better developer, could have known those things up front. While there are definitely things experienced developers know to include in their estimates that inexperienced ones might not (like holidays and sick time, the impact of other work, availability of other resources), what usually kills an estimate is the things that you didn't know when you made it.

Any realistic approach to software development estimates has to account for this. For instance, if I've estimated that a project's going to take me six months, I sit down and re-estimate it after I've been working on it for two or three weeks. I don't ask, "Will the work remaining take me five months and two weeks?" I ask "How much time will the remaining work take me?" If my answer is "Given what I've learned in the last two or three weeks, it still looks like I have six months to go," I get on the phone with the person who I gave the estimate to.

I can't emphasize the importance of re-estimation enough. You can't go back to your original estimate and say "Okay, I've completed the first three things on my list in the time I thought it would take me. Everything's cool." What you've learned has almost certainly changed that list. It's obviously irresponsible to not go back and tell the people who are depending on you (and, most likely, paying you) when your estimate changes. It's just as irresponsible to be unaware that your estimate needs to change.

Answer 19

A good friend of mine works for a consulting company, who has as a client one of the biggest telco's in Germany. The combination of the corporate culture, telco culture, and German culture mean a whole lot of bookkeeping, estimating, and bean-counting. My friend is a scrum master in charge of a few teams of programmers, and he reported to a manager-type person whose job consisted mostly of number crunching and coloring between the lines. Scrum was kryptonite in her world, but since it consistently produced results, management tolerated it.

One thing that they could not stand, however, was the nature of the estimates generated by the scrum teams. Every day she would run the numbers, and discover that the actual number of man-hours required for features and bug fixes was between 20-30% more than the initial estimate. This resulted in many tense confrontations between the two, where she demanded to know why the estimates never reflected the actual time taken for the job.

My friend would simply say, "This is a good thing. If you want to know the actual number of hours, just add 30% to my estimate."

Ok, my story is a bit belaboured here, so I'll get to the point: estimates aren't meant to be trusted. The problem comes when actual important things are scheduled around estimates (which is at times a necessary evil), but it's important to realize that an estimate isn't a piece of hard data, just like how you wouldn't use numbers from the hypothesis in a scientific experiment to prove a related theory -- you'd use the results instead.

But to answer the original question, I tend to think that that estimates are generally taken for granted because people want to believe them. It's like a horoscope... when the predicted outcome is good, people will take it at face value; but when it's bad, they will panic. Even worse, in an environment where estimates are never compared to the outcomes, then people don't have a chance to see the bigger picture, which often re-enforces the psychological impact of the estimate.

So in other words, base your estimates on previous data, not a "gut feeling". Otherwise, you risk entering a vicious cycle of everything being taken for granted, but without good cause to do so.

Answer 20

Maybe you should try to give estimates in ranges. E.g. for this task I will need 7-9 days and try to be honest with those estimates. In most cases you should be able to finish within the given range.

If they ask for exact number of days, then to be sure, for the range 7-9 days tell them 10 days.

The explanation for this reasoning is the similar why e.g. Fibonacci numbers (1,2,3,5,8,13..) are used when estimating user stories. As bigger task is it is harder to estimate it correctly and Fibonacci numbers give you those ranges.

Answer 21

We need to change the way we give estimates. First, don't give days, give hours. Days can translate into calendar days, not '8 hours' which is really what you mean. If you give days, they will mark it down on a calendar starting TODAY. With hours, you can explain that you do not mean contiguous hours nor are you implying a start time. Second, an estimate should be not one point in time, but two. It can be expressed in "10-20 hours" or "15 +/- 5 hours", although the latter may still register in people's mind as one point in time. Also remember that 25% of your time will be spent on non-programming tasks.

Answer 22

I strongly recommend reading the book Software Estimation: Demistifying the black art by Steve McConnell (author of Code Complete).

The most important piece of advice it gave me was the cone of uncertainty.

If you're asked to give an estimate for a task, how much do you currently know about the work to be done?

If it's early, and you know still very little, you give an estimate with a very large range, for example we have no clear requirements therefore it could take anywhere between 1 days and 1000 days (seriously).

Then as you get more information, you can start to narrow this range, ie we now have initial requirements: between 3 days and 30 days.

Just before commencing development you should have enough information give a closer range: eg. between 5 and 10 days. But importantly you never give a single point estimate, the range implies at all times that it is still just an estimate.

This process is called narrowing the cone of uncertainty.

The key point is that it clearly shows management that what they are asking for is just an estimate, not a promise or guarantee. It is completely dependent on actually understanding the problem in enough detail.

Reality dictates that you will not always be given enough time to understand the problem as well as you would like. This process allows you to show the sitauton clearly, alllowing management to then make the call (that is their job).

If they are happy with taking 1-1000 day estimate and using the lower bound (1 day), that is their risk.

By providing an upper limit of 1000 days, you are simply saying that estimation at this point is of minimal value, and that more information needs to be gathered.

But I'm just paraphrasing (probably badly) what Steve says in the book. Go and read the book.

Answer 23

Give a range (e.g. my estimate is between 10 to 15 man days), and give regular updates to the estimate as time goes on.