Software estimation speed and accuracy

Question

Let's say you have a project that will involve two web applications (that will share DAL/DAO/BO assemblies and some OSS libraries):

• a semi complex management application that uses Windows Live ID for authentication and is also capable of communicating with various notifier services (email, sms, twitter etc.), targeted notifiers being about 10% of functionality
• a low to semi complex user application with less functionality but more robustness that also uses Windows Live ID for authentication

There are two of us with medium estimation capabilities and we won't be able to do it in two days even if we wanted/have to. At least it would be a far off estimate.

Questions

1. How long would/does it normally take you to make a reliable/valuable estimate?
2. What would you suggest to speed-up estimation without sacrificing accuracy?
3. How much slack (in terms of cost/time) would you add depending on estimation speed (when you would say: I could estimate it a bit more, because I think it's still quite off)

Answer 1

Since we use Agile methods (Scrum, specifically) it takes us about an hour longer than it takes the users to prioritize.

More time doesn't lead to more accuracy.

The hard part, then, is getting the users to prioritize. We hear this discussion all the time "if the whole thing isn't completed on time, it's all worthless." "Except for the XYZZY component, which does have some value." That argument can go on for hours until it's resolved that XYZZY should be first.

Generally, we try to create 4-week sprints. The first few are complicated because there's always something new. After the first two (or three) we seem to set a steady pace.

Each use case has a relatively simple, subjective valuation of how the effort it will take to finish it. Anything over one full sprint in duration has to be decomposed. Most times a few use cases are bundled into a single sprint.

The are formal ways of scoring each use case to better handle the cost and schedule issues. We don't use them because the extra effort doesn't help.

After the first two sprints,

1. There's new and different functionality,

2. The priorities have all changed,

3. The details of each use case have been dramatically revised.

What does "accuracy" mean when the thing you're trying to estimate changes at the end of each sprint?

---

One lesson learned. Parts of my company spend a long time fully defining exactly what will be delivered, and then measuring that they are delivering precisely what they want.

Customers notice this, and one said we "spend a lot of time delivering what the contract says, but it isn't what we needed."

The problem with firm up-front estimates is that they take on a life of their own. The more you "invest" in the estimating, the more the estimates seem to be a useful deliverable. They aren't useful because they're generally totally wrong. They're based on up-front assumptions that are totally wrong.

It's a bad policy to invest more time in estimating. The "accurate" answers aren't more accurate, but they are more treasured by every layer of management. As you and the customer learn, you invalidate numerous assumptions and you absolutely must re-estimate constantly.

Don't do it up front. If your contract requires you to do it up front, then make sure you have a change control provision and tell the customer that you absolutely will make changes as you go forward. As both you and the customer learn, you both must make changes.

Answer 2

Interesting question. I'm afraid the answer is "it really depends!" I know that's not terribly useful (although it IS true) so here are some factors:

1) Quality and completeness of requirements and their specification. This is, to me, most often the project-estimate killer. If you don't have quality requirements, you have no reasonable basis for an estimate. We use a "RUP-lite" style of product development here, so as the engineering manager I won't give anything but the coarsest-grain estimate until we've completed our "elaboration" phase and gotten sign-off from product management that the core 80% of the product features are in fact accurately covered.

2) The scope & nature of the product. Bigger/more expensive/more complicated = substantially longer to estimate. I've spent years working in telco-carrier land delivering solutions which have the normal robust carrier requirements ("5 9's" of uptime required by SLA mean you must really do a good job of solution design and failure recovery!). In that sort of environment with all the moving parts across functional areas of the business, the estimation of work is going to hinge on getting the whole picture...specifically, cross-functional dependencies and external dependencies can be a REAL killer here. That said, I've also built lots of shrink-wrapped and enterprise software, too. In those environments it's much easier as the scope is typically substantially smaller, so thus easier to estimate.

3) How "new" is this project? How "new" is the team to this product or technology set? The newer the product or team, the longer and more buffer you should allocate.

4) How specific do we need to be? If this is a "rough guess" then I'll lean on my engineering leads to provide a conservative estimate, then I'll pad that. If we need a "real" estimate (e.g., one which is used by my boss and which I'll be responsible for hitting), I'd need the input from a number of different managers and team members, who will need time to analyze the requirements and confer amongst themselves.

That can take as little as a couple days, or weeks..it all depends on the size. "Two or three days" is, frankly, not long enough for sizing anything but the most trivial of projects.

The best thing you can do to improve the quality of your estimates to to improve the quality of your requirements, and be ruthless in identifying hidden dependencies.

One final thing: FWIW, I've been doing this since '81 and I regard accurately estimating a project's duration/cost as the single most difficult/fraught with peril part of engineering management.

Answer 3

To make a reliable estimate you really would need to create a list of tasks to be done. Break it down into stories/tasks (even if you don't use agile) and evaluate them. This can take a lot of time already - especially the amount of research (to look for libraries to do this notifier stuff to reduce the cost). I would take at least 3 days - however 1-2 week(s) sound more reasonable for me. This doesn't seem to be a small project.

I wouldn't dare to speed-up the estimation process if you wan't to have somewhat reasonable results. Estimations are never accurate and you will just make things worse.

An option would be to make a totally rough guess within a few hours and then start to work on it already (for a week or so). At the end of the week you might be able to give a better estimation based on your current progress. However it's important to create a good prototype (which looks like crap but has a little bit of code in all of the regions).

Answer 4

Well, a common quote is "The price will be between 50% and 400% of our initial estimates". The reason that this quote has grown large is that it's true. Estimation accuracy largely depends on your domain-knowledge. If this is the 100th time you sold a given type of blog-engine than you're pretty sure about the estimates. However, more often than not, you don't have that indepth knowledge of the domain (If the app already exists, why create a new one?).

Agile development has become popular because people largely recognize that the traditional "waterfall" type of thinking just doesn't work for most real-world projects. You should apply the same way of thinking to your estimates. Obviously, you need some kind of selling point, but be sure to tell your customers that this information is very vague (and that no matter what any competitor will tell them, their estimates are vague as well).

You need to sell iterations, and therefore also estimate iterations. I'm sure some marketing-guy in any company will know how to handle the paperwork and the legal stuff for this, so it shouldn't be that big a deal.

Consider a 5 iterations project:

• Start out by putting up milestones. These are subject to change, but will provide your vargue first estimates for the final product.
• Plan iteration #1.
• Estimate iteration #1, adjust total estimates accordingly.
• Do iteration #1
• Rinse / repeat till iteration #5
• You're done :)
• Reflect over your project. How did your estimates evolve? Reasons? Learn by doing :)

Most customers would rather have part-estimates and part-deliverances than some unrealistic goal that some suit sold them :)

Answer 5

Straying slightly from your original question but it's also important to learn from the estimates that you have produced in the past by keeping information about how long it actually took to do the things that you estimated.

We estimated 5 days to produce these pages, it actually took 10 days, and etc.

In the long term information like this will (hopefully!) enable you to produce more accurate estimates.