On July 19, 1990, by his account, Arthur Schneiderman gave a talk to a group of executives from about a dozen companies. They had been brought together by Nolan, Norton & Co., a consulting firm, to work out how companies should measure performance. A Harvard professor, Robert Kaplan, was their consultant.
Schneiderman was vice president of quality and productivity at Analog Devices, a maker of integrated circuits. In 1987 he had built a corporate scorecard that put measures of on-time delivery, lead time, yield, defects and cycle time beside the usual financial numbers, many with aggressive targets. Kaplan had found it while writing a case study, and had invited Schneiderman to address the group.
At the end of the talk, as Kaplan told it in a 2010 paper, someone asked how the company was doing. Schneiderman said every quality measure on its corporate scorecard had improved dramatically.
He also noted that Analog’s stock price had fallen by nearly 70 percent in the previous three years.
A cockpit with one dial
Kaplan later put the question they had set out to answer this way: “Why are financial measures alone unable to capture the value-creating activities of contemporary organizations?” Financial measures report what last quarter’s decisions produced. They say little about whether the factory is getting better, whether customers are coming back, or whether people are learning what next year will need. Judging a company by its financial results alone is like driving by the rearview mirror.
Kaplan used a cockpit to make the point. Imagine, he wrote, a plane whose only instrument shows the destination and the scheduled arrival time. The pilot’s incentives might be perfectly aligned with the passengers’, and few of us would want to fly in it.
Analog had added the missing instruments. What its scorecard lacked, Kaplan concluded, was a link from them to what customers valued, or to any customer outcome. The company had not turned better manufacturing and delivery into more sales or higher margins, and that missing link, he wrote, “likely contributed” to the fall. So when Kaplan and Norton wrote up the group’s work for the Harvard Business Review in January 1992, under the title “The Balanced Scorecard: Measures That Drive Performance,” the scorecard looked at a company from four perspectives: financial, customer, internal business process, and a fourth they later named learning and growth. The article opened with a sentence: “What you measure is what you get.”
The man who built it read it differently
Schneiderman, for his part, wrote that he was unaware of any major element of balanced scorecard theory missing from Analog’s scorecard before Kaplan arrived. In 1999, in an article titled “Why Balanced Scorecards Fail,” he plotted Analog’s share of late shipments against its stock price over seven years. He found what he called a good correlation, running the wrong way: as delivery improved, the stock fell. After 1993, delivery got worse and the stock climbed to a split-adjusted peak of around $100 in 1998.
He offered two explanations, the semiconductor business cycle or a lag of more than five years, and admitted that deciding between them required analysis that remained “unconvincing.” Management, he wrote, had to take the link between the two halves of the scorecard on a “leap of faith.”
That is the honest limit of the idea. The non-financial measures are bets about what will show up in profit later. They can be good bets and still be swamped by a market, or take years to pay.
The quarter that looked fine
Imagine you manage a gym, and revenue is up 8 percent this quarter.
Your own card has three more lines. Members who came in at least four times last month: down from 61 percent to 52. Machines out of service for more than a week: up from two to five. Trainers certified on the new classes: one of six.
The revenue rise came from a price increase in January. The other lines describe the members who will decide in December whether to renew. You won’t know for months whether the visits line predicts renewals at your gym. You can start watching it now.
Grading a model on one number
AI models are often ranked by one headline figure, usually accuracy on a benchmark.
In 2022, a team based at Stanford’s Center for Research on Foundation Models published HELM, the Holistic Evaluation of Language Models. They measured seven things across 16 core scenarios: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. Before HELM, they reported, models had on average been evaluated on just 17.9 percent of those core scenarios.
Some of the extra measures told a different story from accuracy. On one reading-comprehension task, NarrativeQA, a model that ranked third on accuracy fell from 72.6 percent to 38.9 percent when the researchers added perturbations such as typos. On one commonsense task, more accurate models were better calibrated; on a similar one, they were worse.
You can build a small card for your own use. Take fifty real examples from your work and score a model’s accuracy. Run the same examples again with typos and rephrasing. Count how often it was wrong while sounding sure, and note cost and time per answer. Run each part a few times, since answers vary from run to run. Which line matters most depends on what you are trying to do, and that is the question a strategy map is for.
Almost in the annual report
Analog had once come close to showing all of this to investors. In 1988, Schneiderman wrote, chief executive Ray Stata published the five-year non-financial scorecard in the Sloan Management Review. The company came within days of printing the whole scorecard in its annual report.
It pulled it at the last minute, he wrote, out of concern that “Wall Street” was not ready.