It isn’t just developer who underestimate tasks, it’s everyone and it’s all the time #HoskWisdom
When we estimate tasks, we underestimate the time it takes because we overestimate our abilities and believe our future involves fewer problems. Most estimates assume no problems, mistakes, other activities will distract us. Using the outside view (similar examples), can help avoid this bias.
This problem is not confined to developers, a common example is people underestimating how long it takes to commute to work and who are late.
Why do developers under estimate?
Happy path estimate — no meetings, problems or any other activities will impede development
There is an unwritten rule when asking a developer to estimate how long it will take, add at least 25 percent onto the estimate or 50 percent if it’s complex. I have seen spreadsheets used with a column for confidence or unknown as a percentage. This column is then used to multiple their estimate and get towards a more accurate estimate.
You are probably thinking this sounds like padding or giving extra time but consider how many times you managed to do a task without the following happening
- Being interrupted
- Finding extra tasks missed in your original estimate
- Extra requirements or scope increasing
- A technical problem
- Finding problems in other peoples code
- User error, adding bugs
- Someone asked you a question or for help on a different problem
- A meeting over running or a new meeting arranged
- unforeseen consequences of other functionality or the whole solution
There is pressure to estimate low from the manager, the customer and the project plan.
If life was perfect, we would get the task, know everything about it, find a quiet place, do the work with no distractions. 99% of the time that happy path doesn’t happen and other activities not on the plan slow us down.
Estimates are not commitments, they are a guide to how long we think a task will take. I have seen developers 99% finished on a piece of development and the last 1% take as long as the first 99% 🙂
“The reason is most people think of themselves as different, and better, than those around them” Daniel Gilbert
We have an above average view of our abilities. If you ask a developer if they are an above average developer, most will reply yes.
Answer yes or no to the questions below
- I am an above average developer
- I am an above average driver
- my intelligence is above average
- I am better than most at picking films to watch
If you answer yes to 3 or 4 of those you believe you are above average but statistically you are average. We believe we are above average and because we are doing it, it will go better. We overestimate how fast we will complete the work.
Another contributing factor is known as the illusion of optimism, we believe good events are more likely and bad events like likely to us. The larger the he task the greater the chance of delays.
The inside view
Making predictions using the inside view is to estimate how long a task will take using the information we have, our understanding of the problem and solution. The brain allows us to to do this fast and make an estimate we believe is accurate.
Making predictions using the inside view risks missing complexity we hadn’t considered or ignored.
The outside view is to consider the task based on statistics or how long it took similar tasks (not the estimate).
Daniel Kahneman gives an example where he and a group of people set out to write a curriculum text book, the group estimate was 2 years. When Kahneman asked a member of the team who was a curriculum expert (who had seen many text books created), how long it took other teams to create a curriculum text book the answer was different (see the quote below)
He fell silent. When he finally spoke, it seemed to me that he was blushing, embarrassed by his own answer: “You know, I never realized this before, but in fact not all the teams at a stage comparable to ours ever did complete their task. A substantial fraction of the teams ended up failing to finish the job.”
This was worrisome; we had never considered the possibility that we might fail. My anxiety rising, I asked how large he estimated that fraction was. “About 40 percent,” he said. By now, a pall of gloom was falling over the room.
“Those who finished, how long did it take them?”
“I cannot think of any group that finished in less than seven years,” Seymour said, “nor any that took more than ten.”
I grasped at a straw: “When you compare our skills and resources to those of the other groups, how good are we? How would you rank us in comparison with these teams?”
Seymour did not hesitate long this time.
“We’re below average,” he said, “but not by much.”
The story is from here Daniel Kahneman: Beware the ‘inside view’ and is an excerpt from the book Thinking, Fast and Slow. It highlights how the inside view can underestimate the time it will take and the outside view can give you an unbiased estimate.
You can watch a video of Daniel Kahneman describing this process here
Can you guess how long it took to create the book? The group (inside view) estimate was 2 years, the outside view between the minimum 7 years — maximum 10 years.
It took 8 years, 4 times the groups original estimate and within the range predicted by the outside view.
How to overcome this?
We are optimistic about our own abilities and estimates of how long it would take us but we are more accurate if we estimate how long it would take someone else (.e.g. an average person).
When you have estimated some work, get another developer to estimate how long they think you would take to do it or ask them how long it took them to do it on a different project.
Compare the estimates of similar developments on other projects or on your project. The time took to do similar developments is devoid of bias and will give you a base rate.
We think we are different and our project is different but they aren’t, so it get an outside view from someone who can step back from the project and asses it analytically.