i

Description

Vue aérienne d'un lotissement de maisons presque identiques le long de rues sinueuses.

Source

Derek Jensen (Tysto) · original page

Licence

PD Public domain

Original photo, unretouched. The background is a version of it stylised by the site.

Reference class forecasting

Before describing what makes your project unique, ask how similar projects ended up.

In brief

• What : estimate based on the distribution of outcomes from similar projects (the “outside view”), not on the details of your own plan (the “inside view”).
• Why : the inside view is systematically too optimistic; the outside view measures what happens on average, including unforeseen events.
• How : choose a reference class, collect actual outcomes, derive a distribution, position your project, adjust cautiously.
• Limit : everything depends on the chosen class and data quality; without comparable historical data, the method does not apply as-is.

On this page

1. Definition
2. The principle: why it works
3. How to do it: the step-by-step procedure
4. Examples
   1. Israel's school textbook roughly the 1970s to 1980s
   2. Cost overruns in major projects 2002 to 2023
   3. Estimating an application redesign Illustrative example
5. When to use it, when to avoid it
   1. Use it
   2. Avoid or adapt it
6. Limits and criticism
7. With AI
   1. Detect
   2. Fix
   3. Mitigate
   4. Prompts to use
   5. Precautions
8. Sources and verification
9. See also
   1. Biases this method fights
   2. Complementary methods

Definition#

Illustration inspired by the person

Fictitious illustration

Fictitious illustration

Reference class forecasting (reference class forecasting) consists of predicting a project's outcome by comparing it with a set of comparable completed projects, then deriving a range from that comparison. It is based on the distinction proposed by Daniel Kahneman and Dan Lovallo (1993, Management Science) between two ways of looking at a project:

• the inside view: you analyze the project itself, its tasks, resources, foreseeable obstacles, and build a scenario;
• the outside view: you treat the project as one case among others in a category, and look at how cases in that category end.

Kahneman and Lovallo described the method in psychology; economist and megaproject researcher Bent Flyvbjerg later developed and spread it for major infrastructure projects. His article “From Nobel Prize to project management: getting risks right” (Project Management Journal, 2006) lays out the procedure. In the UK, following the Treasury's Green Book revision (2003), the Department for Transport and the Treasury decided in 2004 to apply the method to major transport projects, using cost uplifts based on historical data.

The principle: why it works#

When planning from the inside, you describe a sequence of steps that go smoothly. By definition, unexpected events are not in the scenario. Result: you forecast the best plausible case and treat it as the average case. This is the mechanism of The Planning Fallacy, worsened by optimism and by Anchoring on the first number heard.

The outside view short-circuits this mechanism. It does not require guessing unexpected events: the history already contains them, including those we would not have known how to imagine. It also shifts the question from “what will happen to us?” (where everyone sees themselves as special) to “what usually happens?” We now know the opposite bias, just as human: believing you're an exception, which is exactly the tendency this method seeks to correct.

How to do it: the step-by-step procedure#

Flyvbjerg (2006) describes three steps (reference class, distribution, project position). Here is a version adapted to team estimating. Duration: half a day the first time, 30 to 60 minutes once the data is in place. Participants: one or two estimators and someone who maintains historical records (PMO, quality, finance).

1. Estimate first, privately. Each person writes down their own estimate (the inside view), so it can be compared later. Do not show this number before doing the exercise.
2. Choose the reference class. Projects that are similar enough (same type, same order of magnitude, same technology or context) for comparison to make sense, and numerous enough to produce a distribution. This is the delicate step: too narrow and you only have three cases; too broad and the average no longer means anything.
3. Collect actual outcomes. For each project in the class: the initial estimate and the final outcome (cost, duration, delivered scope). What matters is the relative gap between the two. Use end-of-project figures, not memories.
4. Build the distribution. Calculate median, quartiles, and extreme values of the gap. Look at the tail: a small number of heavily off-track projects matters more than the average suggests.
5. Position your project. Place it within the distribution: is the project typical or extreme? The starting point is the class median.
6. Adjust sparingly. Shift the estimate only for specific, verifiable reasons that projects in the class did not have (a team that has already done exactly the same thing, a genuinely different tool). Write down the reason. An adjustment of “because we're better” is rejected.
7. Choose a risk level. A budget or timeline is set on a percentile of the distribution (for example, a cost exceeded only in a minority of cases), chosen based on what you can tolerate if you're wrong.
8. Record the outcome for next time. Every completed project enriches the class.

Deliverables: the class definition and its criteria, the gap distribution, the final estimate with its range, the list of justified adjustments.

Examples#

When to use it, when to avoid it#

Use it#

• For any quantified commitment: duration, budget, effort, before communicating it.
• When the sponsor or client already has a number in mind: historical data is a counterweight (see Anchoring).
• For projects of a recurring type (redesigns, migrations, deployments, developments of a given size).
• To revise along the way: a project that is late relative to its class has a known probability of still being late.

Avoid or adapt it#

• Truly unique project: no sensible class. You can broaden to projects “from the same family of difficulty,” with caution.
• No historical data: you can start with external data (public benchmarks, sector studies), keeping in mind they describe other contexts.
• Contaminated class: if historical figures were “adjusted” during projects (budget revised, scope reduced), the distribution is too flattering.
• To decide on a specific risk: the method gives frequencies, not causes; complement with The pre-mortem.

Limits and criticism#

• Choosing the class is a judgment call. Two honest estimators may choose two classes that produce two ranges. You need to write down the criteria and stick to them.
• Data is scarce and biased. We often only have completed projects (see Survivorship bias); abandoned projects are missing from the statistics and make the historical record look too favorable.
• Adjust too much, and you fall back into the inside view. In practice, teams reopen the discussion to return to their number. The safeguard is to require a written, verifiable reason.
• Past improvement is not guaranteed. An organization that has genuinely improved is penalized by older history; the reverse is also true.
• No cure for political pressure. Flyvbjerg insists that if people benefit from underestimating, the best method will not be enough: you also need transparency and sanctions for manipulated estimates.
• Handle figures with care. The cited averages (28%, 27%) mix sectors, countries, and periods; they show the direction of the bias, not a coefficient to apply as-is to your project.

With AI#

Prompts to use#

1. Build the distribution from historical data. Use with real data from your completed projects (anonymized).

Here is the history of completed projects, with type, size, initial estimate, and final outcome for each: [table].
1. Calculate for each project the relative gap (final - initial) / initial, for duration and for cost.
2. Give the median, quartiles, minimum, maximum, and the number of projects exceeding more than 50% and more than 100%.
3. Flag projects that are not comparable to the following one (describe it): [description of the project to estimate].
Use ONLY the data above: do not add any “market” statistics, and say when the sample is too small to conclude.

2. Review an adjustment. When the team wants to deviate from the class median.

The median of the reference class gives [value]. The team proposes [different value] for these reasons: [list].
For each reason: say whether it is verifiable, whether class projects also benefited from it, and what evidence could be requested.
Conclude: which adjustment is justified, which is not, and why. Do not try to please: if the adjustment looks like optimism, say so.

Precautions#

• Do not ask AI “how long do projects of this type generally take?” while expecting a reliable number: the answer will be plausible, not sourced.
• Provide unfavorable data as well as everything else; cleaned-up history produces a flattering distribution.
• Do not share budgets or client data with an unauthorized tool.

Sources and verification#

• Kahneman, D. & Lovallo, D. (1993), “Timid choices and bold forecasts: A cognitive perspective on risk taking”, Management Science, 39(1), 17-31 — inside view and outside view.
• Flyvbjerg, B. (2006), “From Nobel Prize to project management: getting risks right”, Project Management Journal, 37(3), 5-15 — method procedure.
• Flyvbjerg, B., Holm, M. S. & Buhl, S. (2002), “Underestimating costs in public works projects: error or lie?”, Journal of the American Planning Association, 68(3), 279-295 — 258 projects.
• Flyvbjerg, B. & Budzier, A. (2011), “Why Your IT Project May Be Riskier Than You Think”, Harvard Business Review, September 2011 — 1,471 IT projects.
• Kahneman, D. (2011), Thinking, Fast and Slow, Farrar, Straus and Giroux — chapter 23, “The Outside View”: the school textbook story.
• Flyvbjerg, B. & Gardner, D. (2023), How Big Things Get Done, Currency — general-public synthesis of Flyvbjerg's work.

See also#