What can I do to get better at estimating how long projects are going to take?

Question

I don't want to make life hard for management. I really don't. They're nice enough guys, but every time I am assigned a new project or task and get asked "how long do you think it will take to do this" I end up sputtering off ridiculous timeframes; "between a day and three weeks".

I would like to believe that it's not entirely my fault: I am the company's sole programmer, I am relatively new to proper development (Was it only six months ago that I wrote my first unit test? sigh...), and I am working with a code base that at times is downright absurd.

So I would like some advice. Obviously, experience is the biggest thing I lack but anything that would make me better would be greatly appreciated. I am looking for reading material, methodologies, maybe even actual tools. Any way that I can give my boss more accurate information without having to sit down and design the darn thing first is appreciated.

Ok magic stackoverflow genie, what have you got for me?

EDIT:

@Vaibhav and others suggesting I take time to research and sketch out the system

I agree with you in principle, but how do you balance that with real-world constraints? When you're a one man show or even a part of a small team "I will need 2 days to build an estimate" is a real deterrent when you can fight 4 fires in the time it takes to get a simple estimate.

Answer 1

You need to ask for some time before you give an answer back to management. Go back and study the task:

• Write down an overall approach
• Maybe do some exploration of the code for a few hours
• Break down the task in steps in an excel (I would suggest Microsoft Project, but that may be an overkill)
• Against each task try to assign your best guess of estimate
• Add them up and add a 20-30 percent buffer and give to Management

Over time you will get better.

Update:

Well, it doesn't have to be 2 days. You basically don't give an answer right away. There is no way you can know (except for the most trivial tasks) how much effort it will take right away. If it's a large task (maybe 3-4 weeks) it will take you a couple of days to estimate it, but if its a task which is a 3-4 days, then it will only take you 3-4 hours to break it up and estimate.

Answer 2

Notwithstanding his caveat that it is now obsolete, Joel's article on Painless Software Schedules is extremely helpful if you are new to the concept.

Answer 3

No matter how good you think you or your team is, always overestimate.

Answer 4

Try evidence-based scheduling.

Answer 5

A famous engineer once told me that a good engineer doubles his estimates. If he finished early, then he gets a reputation as a miracle worker.

Answer 6

Overestimate and break down the project in multiple small tasks wich can be devided in either 15 minutes, 30 minutes, 1 hour, 2 hours, 4 hours or 8 hours; If it doesn't fit in 8 hours split the task up again. This should give a fair idea of how long it should take.

Answer 7

Mostly it's just a matter of practice.

Here are some major things to remember when estimating:

1. Use some method of keeping track of how long you thought it was going to take vs. how long it actually took.
2. Make sure to break your tasks down into the smallest possible steps before coming up with your estimate.
3. Don't forget to include testing/bug-fixing time in your estimate.
4. Don't let people force you into artificial schedules.

The Joel article that harpo mentions is a great expansion of these ideas.

Answer 8

You can try reading these two books, they are both great (I read them more than once)

• Software Estimation
• Rapid Development

And they deal also with the human aspect in software construction (which seems may be useful to you)

Answer 9

There is some pretty good written material out there on the topic of estimation. Steve McConnell's books (previously mentioned here) are excellent, and I would also recommend Mike Cohn's Agile Estimating and Planning.

I studied this book before I planned and estimated a large project with my team 18 months ago, and it was tremendously helpful. The approaches and techniques in the book are not just for large projects, though - they are useful for any size of task.

Answer 10

I don't think there is a universal step-by-step prescription.

Every project is different. If you know your code, and you know your processes, then you know enough to do a serious estimate. Just think through all the steps, all the components, and be realistic. Don't be overly optimistic. Build in contingencies for all the places where there is potential risk. It is far better to be accurate than to be fast.

Like anything else: you get better at estimating by doing it a lot. The more you force yourself (or are forced) to make estimates, the better you'll get at doing it.

Answer 11

To get better at making estimates, you need to compare your previous estimates against the actual time it took and then look for patterns in the difference.

As others have suggested, the first step is to break the project down as far as you can so you're working with manageable discrete tasks. Record your estimates for each task and then, as you do the work, record the actual time it takes (including the time for all the bits you forgot about when making the estimate - they're often where the biggest problems lie).

At the end of the project, compare your two sets of figures to see where you went wrong (because, even after years of experience, we all go wrong somewhere). You can then look to improve in two ways:

• Try to spot where you're consistently underestimating or forgetting tasks and then take that learning forward to your next estimate - ie try to improve the accuracy with which you estimate each task. The more you work on similar projects each time, the easier this is to do.

• Also look for an overall pattern in the difference between the total estimate and actual time taken. If you're consistently taking, say, 50% longer than you estimated for each project, you have a guideline for increasing your future estimates. So even if the accuracy of your individual task estimates doesn't improve, you should have a more realistic overall esimate.

You can benefit from the first approach after one project - the second approach becomes more effective as you have more figures to compare.

I also find the second approach to be more helpful if you need to estimate the time other developers will take. I find it hard to estimate on behalf of another developer in a team, but I know that certain people consistently take more or less time than me on particular types of task. So rather than try to work out how long they'll take, I estimate how long I'd take and then apply an overall multiplier based on previous projects. (Ideally, of course, I'd get them to make their own estimates, but this isn't always possible.)

Answer 12

Estimating is hard. Working with NO estimates is a pipe-dream. So it's sort of a necessary evil.

• Start by making it clear that initial estimates are way off so if you come to a X Unit estimate, give a range 0.6X - 1.6X.
• Then there is the trick of doubling the estimate (however saying a ~2 year will take 4 years might be a problem)
• (There's one more involving scaling the magniture and bumping to the next higher unit. Need to refer to my book for this one.)

Next comes the part of how do I reach that estimate so that I can double/multiply? No easy answer.

• Historical trends make sense if you're doing the same kind of project with the same level of team skills.
• Next is the 'Let me steer the ship for 2-3 iterations' and then I'll get back with a better estimate.
• Finally comes the "Tell me how much it will cost.. Till then no one works on this." In this case, you have no guardian angel. Sit and break it down and then apply piecemeal guesses till you have a cumulative estimate. Then do the double-n-bump.

In my 5 years of living in the trenches.. The only thing I've seen is 'guessing by the seat of your pants' estimation... No FP.. No process.. helps in real-world constraints. Still things can improve.. for that I'd suggest getting Mike Cohn's Book 'Agile Estimating and Planning' - that's the book that is closest to us sinful programmers and reality. The rest are preachers on their high horses... Caper Jones which planet is your abode?

Nice question.. I keep asking this when every project begins and then once every week i'm doing the GRIND. Still no answer from the voices in my head.

Answer 13

You should listen to the stackoverflow podcast and hear Jeff Atwood's estimation process for this site you are on right now. He was a little too aggressive and Joel called him out on it.

Answer 14

Also, since it sounds like you don't work a purely software company, keep in mind that nowhere near 100% of developer's time is spent on coding. Make sure you put in time for planning, documentation, research, testing. These are usually the steps that managment would like to cut out of the process, but they won't be happy with the results if they do. If you get any backlash you should remind them, that A) You will end up spending time on those items anyway, whether they are budgeted are not B) You will end spending more time on maintaining and fixing bugs without proper planning. If you haven't read "The Mythical Man Month" by Frederick Brooks, I would hightly recommend it. It has a lot of good concepts and you'll probably relate more than a little to some of the scenarios. Some of the items are a little dated, but there's a lot of good info and you should be able to get one on the cheap.

Answer 15

Try PSP/TSP

Answer 16

These are good points to consider, but you'll need to weed out things that don't work or apply. Revisit the suggestions after you do more estimates, and see if anything stands out as more applicable after using these suggestions.

Keeping track of what you estimated and then how long it actually took is a GREAT idea, and one too often not followed up on, because you need to move on to the next "fire". But, you can only get better at estimating if you spend the time to determine how your estimate was off, and figure out what to change for the next time.

You also mentioned you are working with existing code. As you make more changes to this, I would suggest increasing the time needed to test. You need to make sure the change works, and you need to make sure the other functionality of the code still works. If your tests are somehow automated, then you may need to change the to test this change to the software. If your tests are manual then you need to be sure to test what you changed (adding or changing how you tested previously).

Good luck.

Answer 17

Ask your boss what decisions s/he is going to make based on the estimate you give.

Estimates are decision support. What most people really want are better decisions.

Maybe we can get that by improving our estimates (and there have been a lot of good suggestions given) but, let's face it, estimates are never going to be as accurate as the average PM wants, nor as low as the average salesman wants.

All estimates are ranges and probabilities (even when we're forced to give a single number), and you can provide a lot better decision support if you know what the estimate is going to be used for.

There's a lot more to be said about this, but I wanted to get the perspective out there as no one else has mentioned it.

Answer 18

Don't estimate. Instead, use a project management technique that's not reliant on estimate. Scrum, for example, stresses doing work rather than talking about doing work (ie estimates).

If you spend a day or two or 5 or whatever estimating, wouldn't you be better off having coded or designed or tested on those days, instead of spending the time "estimating"?

Answer 19

Remember Hofstadter's Law:

It always takes longer than you expect, even when you take into account Hofstadter's Law.

It's funny and all, but I actually usually make reasonable estimates, then put some (at least) 10% on top of it and then round it all up. In occasions I was not allowed to do it, I regret to say I ended up failing miserably in keeping up to schedule.

Answer 20

There are a lot of good suggestions on how to do the estimate but they all take time. To get you the time to do an estimate correctly I use this tactic.

When I’m asked for an estimate with out being given time to figure it out how long the project will take I always come back and give an estimate I know management will not like. Let’s say you think a project will take about 3 months. I will say something like 6 months to a year. They usually give me a bad look and I say, “Well, if you let me do a real estimate I can probably give you a better estimate.”

Also, NEVER say a project will take 2-3 months! You are thinking the project will take 3 months and your manager will only hear 2 months.

Answer 21

This may sound glib, but it actually does help. I have had a couple of internal customers over the years who would burst into my office with a new idea. "How long would it take to write a new feature for our big program that does [extremely vague description]?" I found that asking for more details was often futile. "Just give me a rough estimate," he or she would demand.

So I would reach into my desk drawer, and pull out a six-sided die. I would shake it up in my hand and toss it on the desk. "Four weeks," I would pronounce.

"C'mon, be serious," the customer would complain.

At that point, I would explain that without at least a coherent idea of how this new feature would work, I had absolutely no idea, and rolling the die was as good as any other method of estimating. Ultimately, they all got the hint. Most ideas went away to die. With the rest, my customer would come back with something I could work with.

In a nutshell, what I was doing was illustrating the absurdity of trying to come up with an estimate with virtually no information. A picture is worth a thousand words, as they say.

Answer 22

George, I've spent most of my career as a one man team, so I know what you are talking about.

Somebody suggested you get MS Project, if you happen to have a lot of experience with Project, that might be ok. But if not, IMHO it will just make things more complicated.

My first real insight into estimates came from Painless Software Schedules by Joel Spolsky. Joel now suggests you use Evidence Based Scheduling instead, which I believe is in thier bug tracking software FogBugz. I haven't used it, but I'm sure it's a hell of a lot better than what I'm about to tell you ... but if you want to go down the DIY road:

What you need is a simple way to manage a list of tasks. Personally, I added 4 custom fields to my bugtracker software (I use BugTracker.NET), a minimum, maximum, probable, and actual estimate times. I add all tasks and features to bugtracker along with my min, max, & probable estimates. When I finish each task, I update the Actual time. As I move through my tasks, I can see from my history that on average, my 'probable' estimates are off by X% (for me, it's 26%), so I can take my total time, and project all remaining tasks with my min, max, and adjusted probable time. Then on future tasks or projects, I can go through my history of similar tasks, and thier actual times, and form my estimates from those. Having the history will also help avoid missing tasks.

If you do this, you may also run into political BS. When I started this, I gave my manager access, showed him how it works, and how he could set my priorities and figure out the projects progress himself. However, because the software is a bug tracking tool, I believe all features, enhancements, and tasks are being reported as bugs! This is one thing I wish I could undo.

And the Steve McConnell book is fantastic.

Answer 23

i'm late in the game for this question, but your comment struck a chord: you are often asked for estimates on the spot - no time to research, detail, plan, consult with the team, etc. Just a go-with-your-gut number - and probably a number that will come back to haunt you.

This happens - frequently. The trick is to trust your instincts but also compensate for your errors.

In other words, take your best guess and multiply it times some factor that helps to cancel out the lack of information, time to do a proper analysis, and eternal hubris and optimism.

For example, if your last 3 estimates were:

• 1 day
• 2 weeks
• 3 months

and the actual time was:

• 4 days
• 2 months
• 1 year

then clearly your "coefficient of optimism" is 4.

As you get better at estimating - or avoiding estimating, as the case may be - this coefficient should trend downwards. But it may go back up again for a while if you start working in a different domain or technology or team, etc. So it's important to always track it, at least mentally.

For a brand-new area with little info and unfamiliar technology in extreme circumstances - such as repairing the hyperdrive on a Hutt battle cruiser in the middle of a firefight when you can't read Huttese - it's best to employ the Scotty Rule: times two and add a few ^*

[^* a friend of mine insists that the true Montgomery Scott Rule of Estimation is to take your best estimate, round up, multiply it by 10, and change to the next higher order of units. So if you think a task will take 2.5 hours, you round up to 3 hours, multiply by 10 to get 30 hours, and change the units from hours to days to get the final estimate of 30 days.]

Answer 24

I would recommend to split task as much as possible before performing any evaluation, this exercise only will force you to have a deep understanding of what you have to do, and how you plan to do it. Once this is done, you increase your chances of comparing the subtask to something you have already done and have something closer to reality. If the specification of what you have to do is unclear, you will also find out at this moment, which is a good thing.

Tasks over 40h can hardly be evaluated with precision, if your project manager is good, he'll know that! It's ok to be wrong, experience is the only real thing that will make a real difference in your evaluations (and in the end, will only help you be closer to a correct answer, not perfect).

I also had a very nice experience with poker planning, the more my team uses it, the better their evaluations (plus it's fun!) but it takes more than 1 person which can't help you much.

Answer 25

I used many but maybe somthing like the 3-point estimating tool might be worth a look at.

3 Point Estimating Tool

With this you take the Optimistic (if everything went to plan), Mostly Likely (how long you think the project will take) and the Pessimistic (longest you think it could take)

You can then work out a Mean and add a weighted average to parts as well if needed. The more you break you project down the better the esitimate you might get.

As a quick start, you could have something like this as your main headers, with any numbers of tasks in them, with there Optimistic, Most Likely and Pessimistic ratings

• Research
• Function Spec
• Design Spec
• Coding
• Testing
• Deployment

Of course you might want to expand on some heading, or add you own.

Answer 26

Something I use is COCOMO. It's a technique that provides probabilistic models for software cost estimating. It's something that your PMs (if formally trained) will likely be familiar with. Instead of being concerned about hours, you simply estimate the SLOC (source lines of code) you expect to write. That right there should set off some alarms, but it's only one piece of the technique. You then plug these estimates into a model and adjust the metrics for fine tuning.

• Number of developers
• Capability of each developer
• Your work calendar (hours per week, per resource)
• Development language(s) & percentages (if using more than one)
• Familiarity with the domain (has anyone written this type of app before)?
• Management style
• Development process (Agile, waterfall, RUP, ...)
• etc.

What this whole process does is let you focus on what you're good at estimating: code. The time estimate is abstracted and encapsulated by the model itself. Each model has been constructed by analyzing 100's of real-world projects. You can even tune them yourself, by plugging in your own actuals.

I've had a lot of resistance to this method, especially when I forget to round. "Oh, right. This will take 29.02 hours." All in all, it's been the most accurate method for me. My colleagues can disagree with the usage of SLOC all they want, but they cannot debate the science that went into creating this technique.

As a sole developer, I have used it, but had problems with my ego at the beginning. I didn't want to acknowledge that my C++ skills were weaker than I thought at the time. It caused me to underestimate. After I plugged in my actual skill levels, everything lined up. Lesson learned.

Another technique I throw on top (some tools do this for you), is the three estimate technique, which is broadly applicable in all places. You take your worst case, best case and expected case then average them together. Weight your expected by a factor of four. So... "(worst + best + expected * 4) / 6" It's a moderate help for risk, but it's actually a psychological trick to get accurate estimates more quickly... in favor of your gut feeling.

Answer 27

You can try to break it down in smaller tasks, and estimate each task. For example, you could break it into something like:

1. Install SharePoint
2. Create proof of concept web part.
3. Deploy custom code into SharePoint.

These are tasks that you should be able to find tutorials for, and by reading them, you may be able to grasp how long it will take (hint: installing SharePoint is long!).

Besides this, I agree that it's hard to estimate tasks that you haven't done before, so try to break it down to tasks that are small and comparable (in scope) to things you've done in the past.

Answer 28

If someone else in the company has done similar stuff yell over the cube wall and see what they think. Compare that to how long it actually takes you, then when you get another task ask the same developer how long it would take them. Multiply that result by the difference you got doing the initial task. There's really no good way to estimate new tasks with out a historical baseline but at least this way will show you made an effort to base your estimate on some kind of fact.

Answer 29

Double your initial estimation.

Seriously though, try to break up the task into smaller, estimate-able bits. And then double it.

Answer 30

Hi,

As a developer who became a project manager I can tell you that this is not easy. The key point is to break the task down into smaller and smaller chunks, estimate each one then add your estimates together. Obviously this may take some time so speak to your PM about how long you have to produce the estimate. Also any good PM will know that an estimate is just that, so keep them informed with your progress, remember ultimately you are all part of the same team.

Hope that helps,

Dan.

Answer 31

estimate quickly, re-estimate regularly and most importantly keep management updated.

It's all about managing expectations, so make sure they understand the limitations of your estimate, this is key, especially for non technical project managers.

Answer 32

This question has a long answer and a slightly less longer answer (but no short answer).

The long answer is 255 pages (and some appendices) long and is called "Software Estimation: Demystifying the Black Art" by Steve McConnell. It comes highly recommended.

The slightly less longer answer is actually just a set of heuristics/tips:

1. Break the task down into small items - preferably 1-3 days each
2. Try to find a similar task - in the same domain or involving similar technologies. It's almost never the case that you're estimating a task that you've done before (because it's already done...). Example: If you're task is "enhance SharePoint in way X",maybe you've done a task that involved integrating with SharePoint and/or another Office product and/or another "Content management system"
3. Give your estimation as a range rather than a single point estimation
4. Understand your margins - if you're estimating something completely new you might need to take a 100% or even 400% factor (i.e. if your estimation is 2 days and the factor is 400%, it could take between .5 a day to 8 days). The more uncertainty, the higher the factor.
5. Understand that as you progress on the task, you need to revisit and refine your estimation
6. Communicate, communicate, communicate! Make sure management is aware of how reliable a specific estimate is (or isn't).
7. Give estimates in a scales and numbers that reflect the accuracy (e.g. estimating in hours implies more accuracy than giving the same estimate in days; Estimating it would take 3.75 days causes people to automatically assume your estimate is more accurate than if an estimate of 4 would have been given)
8. Acknowledge the "Cone of Uncertainty" (see more details below)
9. When giving an estimate, let's say "3 days", think of the following question: "Can it take less than 3 days?". If the answer is No, then this is an optimistic estimate, since it would take 3 days if and only if nothing goes wrong...

Several more notes:

• Inaccuracies can go both ways, either up or down (although software projects are notorious for under-estimates rather than over-estimates)
• Note that the effects of underestimating are usually more detrimental to the project than overestimating (this is also discussed in the first part of the aforementioned book)

---

The Cone of Uncertainty:

The idea of "the cone" is that you start a project with high-level/low-accuracy estimates and your confidence and accuracy increase as you make progress on the project. For example:

• At the time of requirement gathering your estimates are much less accurate than after the high-level design is ready
• Which are far less accurate than after the detailed design was done
• Which is less accurate than halfway through coding.

This is a fact of life, and instead of fighting it, this understanding should be built into your plan. This is mostly relevant when estimating a whole project or a big task (i.e. longer than 2 weeks), since splitting it into fine-grained items like I suggested in item 1 is not really feasible (or cost-effective).

Answer 33

One of the classic techniques in project management is to take 3-point estimates. Basically ask yourself how long is it going to take (or how much effort is required) to finish this task? What are your optimistic, most likely and worst-case-scenario guesses? From what you say you will probably come up very large gaps in your estimates, which indicates risk.

What you need to do is to try and reduce that risk as much as possible. Here are a few things you can do (as you probably already thought of):

• Read about the subject
• Create a spike
• Ask others who has experience (don't be wrongly influenced though!)
• Make sure your project manager knows about this!

After all this estimate again to see if there are any improvements ;-)

Answer 34

The important thing is not giving an estimate that "sounds" OK for your Boss because he's expecting something, just really estimate what you think. If he questions your estimate, just say that you'll keep him updated from time to time and remember him that you've never work with the technology.

Also, I always make two estimates, the pessimistic and optimistic scenario. I ADD 10 to 20% at the total for tests and debugging (normally, nobody will ask you to remove the debugging time).

Answer 35

There's usually an asymmetric penalty function here, for the manager and the dev team. That's actually part of the logic of padding.

Late: If people tell their bosses to expect Tsched, and instead it is longer than Tsched, then schedules get bumped and people get faulted for various things and in the worst cases sales gets missed or customers go away or projects are re-organized or trash-canned.

Early: If the true T is less than Tsched, then hey, pat the dev team on the back and get busy on the next project. Or -- heaven forbid -- refactor and improve the code quality (no, don't do that... do the next project. workworkwork).

Answer 36

Estimate on the estimate.

In many cases coming up with the estimate for a project represents a lot of the work involved in the project. If you can establish the trust of your clients, you can charge a day-rate to develop an estimate. This has a number of benefits for you and for them.

1. The estimate will be more accurate
2. You'll know what you're about to do and they'll know what help you need
3. You can walk away without hard feelings if the project looks too expensive

The standard timeframe for planning on construction projects is 10% of the total timeline for the project. No reason to throw out hundreds of years' learning for a few transistors.

EDIT : For smaller projects this phase is usually where the needs assessment and feasibility study get done for free by a hungry developer .... most of us have been there ....