In December 2007, Barack Obama was trailing by double digits in the polls, and his campaign’s website had a splash page: a picture or a video at the top, and a button to sign up with an email address. Each address could later be asked to volunteer, and asked to give.
The staff knew which video they wanted up there. They “heavily favored” one called “Sam’s Video,” wrote Dan Siroker, the campaign’s director of analytics. “Had we not run this experiment, we would have very likely used that video on the splash page.”
Instead his team built 24 versions of the page, six pieces of media, three images and three videos, times four buttons. Every visitor was shown one at random, and the campaign tracked who signed up. A total of 310,382 visitors went through.
“Everyone on the campaign loved the videos,” Siroker wrote.
Every video lost
“All the videos ended up doing worse than all the images.” In a reply under his post, Siroker gave the numbers: the videos signed up 5.95 to 7.76 percent of visitors, the images 8.54 to 9.66 percent. The best version, a family photo with a button that said “Learn More,” signed up 11.6 percent. The original page had managed 8.26 percent.
Build a little, measure what people actually do, and let it change your next move. Eric Ries, author of The Lean Startup (2011), calls it the build-measure-learn feedback loop. “The fundamental activity of a startup is to turn ideas into products, measure how customers respond, and then learn whether to pivot or persevere,” he writes on the book’s website. All successful startup processes, he adds, “should be geared to accelerate that feedback loop.”
Throw darts in the dark and you can practice for years, growing more confident and no more accurate. Measuring turns the lights on, one throw at a time.
The campaign’s test changed what came next. “Because this first experiment proved to be so effective,” Siroker wrote, “we continued to run dozens of experiments across the entire website throughout the campaign.”
One loop, a year long
Skip the small loop and you still get a loop, only bigger. When Ron Johnson planned to replace most of J.C. Penney’s sales and coupons with everyday low prices, he was asked about trying the new pricing first on a limited test basis. He reportedly said: “We didn’t test at Apple.”
The plan went out across the chain early in 2012. By the end of the fiscal year, revenue had fallen 24.8 percent. Then came the learning. “We learned she prefers a sale,” Johnson said of Penney’s customer. “At times, she loves a coupon.”
What a test can’t tell you
A loop is only as good as what you put into it. Teppo Felin and colleagues have argued that lean startup gives little guidance on where hypotheses come from, that learning from customer feedback has limits, and that its experiments tend to produce incremental results. That is how Nancy Bocken and Yuliya Snihur summarized the critique in 2020, before answering that lean startup was built for iterative experimentation, not for coming up with ideas.
The Obama test picked the best of 24 versions that people had already made. It couldn’t invent a 25th. It measured sign-ups, the thing it could count. Siroker’s estimate that the winning page brought in $60 million more rests on assuming the improvement held for the rest of the campaign, as he said himself.
The call or the text
Imagine you run the front desk at a dental clinic where too many patients miss appointments. One receptionist wants to replace the reminder text with a phone call the day before. Another says the calls will eat her mornings.
So you alternate. The next 80 appointments get a text or a call, one after the other, and before you start, you write down what counts: a missed appointment. The results come back 5 misses after texts, 4 after calls. With 80 appointments, that’s a coin toss, not an answer. What you did learn is that the calls took about 40 minutes a day, so the next loop can be longer, or cheaper.
The drive-thru that went quiet
McDonald’s put AI through the slower version. From October 2021, when it formed a partnership with IBM, it tested an AI system that took orders at the drive-thru, in about 100 restaurants. The goal, according to Nation’s Restaurant News, was to see if automated voice ordering could simplify work for employees and make ordering faster and better for customers. NRN also reported that McDonald’s was aiming for order accuracy of 95 percent or more before a broader rollout.
In June 2024, McDonald’s told NRN: “After thoughtful review, McDonald’s has decided to end our current global partnership with IBM on AOT beyond this year.” It said it had captured “many learnings,” and that the work had given it “the confidence that a voice ordering solution for drive-thru will be part of our restaurants’ future.”
Before an AI tool goes to everyone, write down the bar it has to clear, in your own terms, such as 45 of 50 real customer requests handled correctly. Run it on those 50 and check each answer by hand. Run it again, because the same tool can answer differently the next time. Then decide.
McDonald’s said it planned to make “an informed decision on a future voice ordering solution by the end of the year.” The next loop would start from what about 100 drive-thrus had taught it in two and a half years.