The plan, as Edinburgh’s councillors approved it, was a tram from the airport to Newhaven on the waterfront, open in the summer of 2011, within a budget of £545 million. To get there, the council’s own company, Transport Initiatives Edinburgh, known as tie, had to answer a question official guidance now put to big public projects: what usually happens to ones like this?
In its draft business case of May 2005, tie used a 2002 review by the consultants Mott MacDonald for the Treasury, which sorted projects into standard and non-standard civil engineering. tie put the tram in standard civil engineering, saying it did so on advice from the review’s authors. In the review, the projects in that category were all roads.
The choice decided the number. Standard civil engineering meant an uplift of 44 percent, which the headroom between the estimate and the available funding could absorb. Non-standard would have meant 66 percent. Newer rail figures, which the engineering firm Arup had already applied to one of Edinburgh’s lines in 2004, meant 57 percent. Neither would have fitted.
tie told the council it was 95 percent certain it could deliver within the budget.
A tram among the roads
The tram opened on May 31, 2014, almost three years late, and only from the airport to York Place. A public inquiry put the best estimate of its cost at £835.739 million. tie’s former risk manager told the inquiry that the larger allowance for a non-standard project “would just compromise the deliverability of the system” or alarm funders.
Before trusting your plan, check how similar projects actually turned out.
The inside view starts from the project in front of you: its plan, its risk register. In 1993 Daniel Kahneman and Dan Lovallo traced overly optimistic forecasts to this view, which “anchors predictions on plans and scenarios.” The outside view first sets the details aside and asks how a class of similar cases turned out. Kahneman’s own textbook story is told under the planning fallacy.
A satnav works the outside way. It doesn’t ask how fast you mean to drive. It tells you how long the road has been taking everyone else.
The planning researcher Bent Flyvbjerg turned the idea into a procedure he called reference class forecasting: pick a class of past, similar projects, get the spread of their outcomes, and place your project in it. The class, he wrote, must be “broad enough to be statistically meaningful but narrow enough to be truly comparable.” In 2004 Britain’s Department for Transport and the Treasury adopted it for large transport projects. By Flyvbjerg’s account, its first practical use was that 2004 review of an Edinburgh line.
Who picks the class
Edinburgh shows where the method leaks. Flyvbjerg, a co-author of the 2004 guidance, acknowledged to the inquiry that it let users judge where in the class their project belonged, a step prone to bias. tie went further. It cut its uplifts, citing its risk management, and by the final business case in December 2007 there was no allowance for optimism bias beyond a 12 percent risk allowance. Flyvbjerg called that 12 percent “simply unrealistic.”
Flyvbjerg wrote that a class is also harder to find for projects with few precedents, such as new technologies. The inquiry found the guidance built on data almost two decades old, with no class of its own for light rail. And in 2013 the economists Jonas Eliasson and Mogens Fosgerau argued that average overruns can appear even when forecasts are unbiased, because projects are chosen partly for how good their forecasts look. Uplifts, they wrote, help budget a whole portfolio but improve which projects get picked only if classes with known biases can be told apart.
Nor did the forecast alone undo the tram. The inquiry listed ten principal causes, from the infrastructure contract to how the project was governed; the missing uplift was one of them.
Which past projects count
Imagine you’re asked to budget a rebuild of your company’s website. The finance system shows the last five web projects. Three were redesigns and came in within 10 percent of plan. Two also rebuilt the checkout, and ran 60 percent over.
Your plan calls this one a redesign. Its scope touches the checkout. Whichever pile you put it in, the budget follows.
Write down which projects count, and why, before you work out what they add up to. To move yours to the cheaper pile, you’d need a reason you can show. Flyvbjerg told the inquiry that a team’s documented record of beating its class could justify a lower uplift, though he had never seen one that ultimately did.
A base rate from memory
In a 2024 study, a team from Google and the University of Washington had PaLM 2, whose training ended in July 2022, forecast questions from a prediction market run among Google employees. In one setup, the model first posed itself a question about how often similar events had happened, answered it with no data supplied, and then forecast. On the main question set its Brier score (lower is better) was 0.1562, against 0.1221 for a plain prompt. The human market scored 0.1334.
The plain prompt’s lead, the authors argued, came from guessing low on questions that mostly resolved no. When they scored the questions that resolved yes and those that resolved no separately, then weighted the two groups equally, the human market beat every model setup. On the yes questions alone, two setups did better than the humans. This was one older model, and the paper doesn’t report whether its base rates were right, only that the model supplied them itself.
Hand the model the class yourself. Paste in the past projects, ask which ones it counted and what range they give, and recompute one figure by hand. Ask again in a few fresh chats; if the projects it counts keep changing, find out why before using the number.
In 2019 a business case for the last stretch, from York Place to Newhaven, went to the council. This time the estimate, £207.3 million, included an allowance for optimism bias.