Evidence-Based-Scheduling - are estimations only as accurate as the work-plan they're based on?

Question

I've been using FogBugz's Evidence Based Scheduling (for the uninitiated, Joel explains) for a while now and there's an inherent problem I can't seem to work around.

The system is good at telling me the probability that a given project will be delivered at some date, given the detailed list of tasks that comprise the project. However, it does not take into account the fact that during development additional tasks always pop up.

Now, there's the garbage-can approach of creating a generic task/scheduled-item for "last minute hacks" or "integration tasks", or what have you, but that clearly goes against the idea of aggregating the estimates of many small cases.

It's often the case that during the development stage of a project you realize that there's a whole area your planning didn't cover, because, well, that's the nature of developing stuff that hasn't been developed before. So now your ~3 month project may very well turn into a 6 month project, but not because your estimations were off (you could be the best estimator in the world, for those task the comprised your initial work plan); rather because you ended up adding a whole bunch of new tasks that weren't there to begin with.

EBS doesn't help you with that. It could, theoretically (I guess). It could, perhaps, measure the amount of work you add to a project over time and take that into consideration when estimating the time remaining on a given project. Just a thought.

In other words, EBS works on a task basis, but not on a project/release basis - but the latter is what's important. It's what your boss typically cares about - delivery date, not the time it takes to finish each task along the way, and not the time it would have taken, if your planning was perfect.

So the question is (yes, there's a question here, don't close it):

What's your methodology when it comes to using EBS in FogBugz and how do you solve the problem above, which seems to be a main cause of schedule delays and mispredictions?

Edit

Some more thoughts after reading a few answers:

If it comes down to having to choose which delivery date you're comfortable presenting to your higher-ups by squinting at the delivery-probability graph and choosing 80%, or 95%, or 60% (based on what, exactly?) then we've resorted to plain old buffering/factoring of our estimates. In which case, couldn't we have skipped the meticulous case by case hour-sized estimation effort step? By forcing ourselves to break down tasks that take more than a day into smaller chunks of work haven't we just deluded ourselves into thinking our planning is as tight and thorough as it could be?

People may be consistently bad estimators that do not even learn from their past mistakes. In that respect, having an EBS system is certainly better than not having one. But what can we do about the fact that we're not that good in planning as well? I'm not sure it's a problem that can be solved by a similar system. Our estimates are wrong because of tendencies to be overly optimistic/pessimistic about certain tasks, and because of neglect to account for systematic delays (e.g. sick days, major bug crisis) - and usually not because we lack knowledge about the work that needs to be done. Our planning, on the other hand, is often incomplete because we simply don't have enough knowledge in this early stage; and I don't see how an EBS-like system could fill that gap. So we're back to methodology. We need to find a way to accommodate bad or incomplete work plans that's better than voodoo-multiplication.

Answer 1

I've been meaning to ask the same question. The best I can think of is to simply minimize the effect of those forgotten tasks by iterating more often, so that they never account for more than, say, 5% of the scheduled time for each iteration.

But I assume that EBS could help if the following holds true: does each person "miss" tasks in a consistent manner, the same way that each person makes consistently biased guesses for the individual tasks?

Answer 2

If you are talking about scope changes adding new tasks, you should be buffering for this in your schedule

If your talking about a not understanding a problem well enough to define the right tasks before it is estimated, then it can be treated the same as an interruption. Don't worry about these, just keep the clock running and let the law of averages work it out.

I'm sure Joel will have a better answer, but thats my take on it.

Answer 3

This is the problem that has plagued project manager forever (all the way back to the pyramids).

• We missed something in the design/analysis.

The immediate fix is simple, but painful. You have to do a change-request/change-order (who pays for it might be part of the pain). You create your new work items, and your schedule adjusts.

The longer term fix is also simple, but not guaranteed to be pain free (or perfect). You must estimate smaller tasks. Typically I only find these kinds of goofs when a task is estimated in the terms of days or weeks.

• No task should be more than a day or two long, (exception: route tasks where you have multiples of the same task - e.g., I ave 50 forms, and each one needs to have the fonts changed on all the buttons, labels and text boxes -> 50 forms * 2 hours/form => 100 hours)
• No task is less than an hour. I don't care if it is a one line change. You are still going to have to ramp up, ramp down, check out, check in, test, file a TPS report, etc. no mater how small the task is.

What happens if I have a task that I know is going to take a week? You haven't done enough analysis. There MUST be break points in there that can break it down into at least two day chunks.

EDIT
Added Work is simply that. Added work. And as such they become new tasks. EBS (or any other planning and estimating system) is simply not set up to handle added tasks. It has not inherent way of knowing what unknown-unknowns you have. From there I direct you back to the "longer term fix" I mentioned above. If you can't do a task in one or two days, you probably don't know it well enough to estimate correctly.

All you can do to handle unknown-unknowns is include some insurance to the process by adding contingency funds/time. Typically I budget 20% in contingency. So I create a task called contingency when I first start a project. If I come across an unknown-unknown, then I add a task for it, estimate it and decrement the contingency task by that same amount.

Answer 4

This is a difficult one. It may be possible to take the historical original estimated hours : final estimated hours ratio for past milestones and do some kind of reasonable projection of how many hours of work this milestone will take after you're done finding tasks, but as you observe, EBS doesn't do that (yet, anyway).

These are three things that I do to make EBS work for me as it stands now (and sorry for the overlap with other answers here):

1. Break down the milestones as much as possible Sure, there may be uncertainty later on in a project, but I usually am pretty sure what I'm going to be doing for the first 6 weeks or so. So making milestone releases with about 1-2 months of work help me to see if we're on or off for the tasks we know about.

2. If I'm pretty sure there are going to be tasks in a milestone that I just don't know about, I look at the EBS projected 50 and 95 percent dates and set the official release date for the release in FogBugz to accommodate some buffer for that -- looking at the Ship Date Over Time graph for a past release that I think is analogous to this one can be a big help; you can see where you added a bunch of stuff, and how much it affected the projected and final ship dates.

3. If (2) won't work because you need this release's EBS projection to stack up with future releases properly, you're right, right now the best thing to do is make a big ol' 'Stuff We'll Figure Out That We Forgot Later On' schedule item for each developer. If I have to do this, then I usually estimate each new task and subtract that estimate from the buffer schedule item, then just remove its estimate and close it when I think we've found all of the tasks. This is a little bit cowboy, and really proper EBS usage would say you should charge your time for each of these found tasks against the initial schedule item, but hey, this is a place where you may have a better sense of overall uncertainty than EBS does, so I say go for it.

Hope that's of at least some help; I agree that this is one of EBS's big current problems. It's fixable with a little bit of project management art, but hopefully EBS can bring some science to it in the future.

Answer 5

"But what can we do about the fact that we're not that good in planning as well? I'm not sure it's a problem that can be solved by a similar system. "

The "predict the future" problem cannot be solved by tools or process. You simply cannot succeed at this game.

1. You can't win. There's no way to predict the future of a software development project. Software development is knowledge capture. If you already knew everythhing, you'd be doing an install. Since you're building new software, there has to be one or more things you don't know (either user-oriented or technical.)

2. You can't even break even. There's no way to arrive at an acceptable prediction because every change is punished. If you provide a "preliminary" estimate of 6 months, and the users double the feature set, you're in trouble for doubling your estimate. As you learn more about the use cases and technology, and adjust your plans, you're accused of "scope creep".

3. You can't get out of the game. Everyone wants a rock solid, indisputable price for the software product, in spite of the obvious fact that no such thing can ever exist. And "indisputable" is entirely to the taste of the people you're presenting to. Some folks want a price that you can never run over, no matter how huge -- they want the "total" cost of ownership. Other folks want a price that you've only got a 60% chance of meeting -- they just want an initial budget for this year's development effort.

Your only choice is to attempt to use more Agile techniques to build the most valuable thing first with a "pay-as-you-go" model. Estimate a little, build a little, get some income.

In short, you have to change the game to avoid attempting to provide rock-solid, indisputable, total-cost of ownership estimates. Provide a development budget (people cost * time) for the first release and a sequence of releases tied to specific features.

If you build from most important to least important, the buyers can stop spending at any time and have something of value.

Answer 6

It's true that EBS gives you a ship date based on its current list of tasks (and the historical accuracy of your estimates in the past). It sounds though like you want it to predit the future :)

If you discover something you missed in your planning stage, it doesn't seem that ANY piece of software would ever have been able to account for that. If it could, it probably would have just written the software itself, and then the world would be run by robots.

This is an interesting question though because it does come up a lot and every time it does, I become confused again. Maybe it's me just not understanding what you're asking, but you wrote:

"It's often the case that during the development stage of a project you realize that there's a whole area your planning didn't cover, because, well, that's the nature of developing stuff that hasn't been developed before."

How could any piece of software, that doesn't have a "future telling" module, estimate for something that you didn't know about? I don't mean this to be snarky in any way. I'm genuinely interested in the answer because you aren't the first to ask it.

My guess is that perhaps what people expect from EBS (tell me the date I'm going to ship) is different than what EBS promises (given the data you've given me, what's the probability I'll ship on this day). Maybe it's like that CGI movie with the train and Tom Hanks where he looks too freaky real and so the movie is really creepy. Maybe EBS sounds like it can predict the future, so when it starts doing things that seem like that, people want it to actually do that.