On July 26, 2009, two teams crossed the same finish line within twenty minutes of each other.
The race was the Netflix Prize. For nearly three years, teams had been trying to beat Cinematch, the system Netflix used to predict how many stars a member would give a movie. The finish line was a number: cut the prediction error by 10 percent, from 0.9525 to 0.8572 or lower, on a hidden set of ratings. The reward was $1 million.
That afternoon, a team called BellKor’s Pragmatic Chaos submitted predictions at 18:18:28. A team called The Ensemble submitted at 18:38:22. On the hidden test set, both scored 0.8567, a 10.06 percent improvement. The rules had settled ties in advance: the submission received earliest would be judged first. About twenty minutes decided a million dollars.
Nobody had to argue about who won. That was the point of the number. Less than three years later, Netflix explained what it had done with the winning work.
A number anyone could check
Netflix wanted better recommendations. You can’t hold a worldwide contest on “better.” You can hold one on 0.8572.
If you can’t say which number would prove it, it isn’t a key result yet. In an OKR, the objective says where you’re going, and each key result is the number that settles whether you got there. Andy Grove, who built the method at Intel, said in a 1970s classroom video that at the end you should be able to look at a key result “and without any argument say, ‘Did I do that, or did I not do that?’” (The OKR page tells how Intel used them.)
The Netflix number did that job well. In its rules, Netflix guessed 10 percent was “pretty tough” and might take years, so it added a $50,000 Progress Prize for each year’s best improvement. A year in, a team called Korbell won the first with 8.43 percent, after more than 2,000 hours of work and a blend of 107 algorithms. Netflix took the two best of those algorithms, adapted them to its own data of more than 5 billion ratings, and put them into production. None of that could have been judged against a goal of “better.”
Ten percent of what
The Grand Prize blend was another matter. In April 2012, two Netflix engineers wrote on the company’s tech blog that they had evaluated some of the new methods offline, “but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment.”
By then Netflix had changed. It launched streaming in 2007, a year after the contest began. Members now picked something to watch on the spot, and Netflix could see whether they finished it. The engineers gave the business objective as member satisfaction and month-to-month retention. Predicting star ratings was, in their words, “just one of the many components.” They were candid about the original choice: the contest had needed “a proxy question that was easier to evaluate and quantify.”
So the number did two things. It made progress checkable, which is why the first-year algorithms reached production. And it measured a proxy. A key result can be perfectly measurable and still drift away from the result you wanted.
Shipping three things
Imagine you manage search for a shopping app, and your objective is “customers find what they’re looking for.” Your first key result: ship three search improvements this quarter.
That’s measurable, but it measures your team’s activity. You could ship all three and customers could still search, fail, and leave.
So you rewrite it: the share of searches followed by an item added to the cart within two minutes rises from 18 percent to 25 percent. Now the number is closer to the result. It is still a proxy, so once a month you also read fifty search sessions that ended with no cart, to see what the number misses.
When the test is the target
AI coding agents are judged the way the Netflix teams were: a hidden test decides whether the work counts. SWE-bench gives a model a real GitHub issue and a code repository. The model’s fix counts as correct if unit tests from the original fix pass.
In August 2024, OpenAI reported problems with those tests. It had 93 developers screen a random sample of 1,699 tasks, and filtered out 68.3 percent of them, for issue descriptions too vague to solve or tests that could reject valid solutions. The 500 tasks it selected on that basis became SWE-bench Verified, which became a standard number in model launches.
In February 2026, OpenAI stopped reporting it. In an audit of 138 tasks that its o3 model did not solve consistently, it found that 59.4 percent had material problems in test design or problem description. It also found that all the frontier models it tested could reproduce, for certain tasks, the original human-written fix or details of the problem statement, a sign they had seen them in training.
“Tests pass” was, in Netflix’s phrase, a proxy question that was easier to evaluate. When you set one for an agent, write down beside it what the tests don’t check. Keep a few checks of your own that the agent never sees, run them after it says it’s done, and read a sample of the changes that passed.
Still running
By Netflix’s 2012 account, the $1 million blend was never worth the engineering to put into production. The two algorithms it had picked from the first year’s 8.43 percent prize-winning blend were by Netflix’s 2012 account still part of its recommendation engine.