In 2006 Netflix offered $1 million to anyone who could improve its movie recommendations. The contest, though, would not judge recommendations. It would judge a number.
Netflix’s system, Cinematch, predicted how many stars a member would give a movie. The contest measured the error of those predictions, the root mean squared error, on a hidden set of ratings. Cinematch’s error was 0.9525. To win the Grand Prize, a team had to get it to 0.8572 or lower, a 10 percent improvement. In the rules, Netflix wrote that it suspected 10 percent was “pretty tough” and that it “may take months; it might take years.” To keep things interesting, it added a $50,000 Progress Prize for each year’s best improvement.
A year in, a team called Korbell won the first Progress Prize with an 8.43 percent improvement. It reported more than 2,000 hours of work and a final combination of 107 algorithms. Netflix took the two best of those algorithms, adapted them to its own data of more than 5 billion ratings, and put them into production.
The last stretch took two more years. On July 26, 2009, near the end of the contest, two teams submitted predictions that scored exactly the same on the hidden test set: an error of 0.8567, a 10.06 percent improvement. On the leaderboard, BellKor’s Pragmatic Chaos had submitted at 18:18:28 and The Ensemble at 18:38:22. The rules had settled ties in advance: the submission received earliest would be judged first. In Netflix’s words, the winning entry came “just 24 minutes before the conclusion of the nearly three-year-long contest.”
The prize was handed over in New York on September 21, 2009, for an ensemble that blended hundreds of predictive models.
In April 2012, two Netflix engineers explained on the company’s tech blog what had happened to it. “We evaluated some of the new methods offline,” they wrote, “but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment.”
The business had also moved. Netflix launched streaming in 2007, a year after the contest began. By 2012, members chose what to watch in the moment, sampled several titles, and Netflix could see whether a video was watched to the end. The engineers gave the company’s business objective as maximizing member satisfaction and month-to-month subscription retention. The contest, they wrote, had needed “a proxy question that was easier to evaluate and quantify.” Accurate rating prediction was, in their words, “just one of the many components” of a recommendation system.