Back in 2007, Dan Siroker later wrote, when Barack Obama was running for the nomination and trailing by double digits in the polls, a website was pretty much all the campaign had. Siroker was its director of analytics. His job was to use data to help the campaign make better decisions.
The site had a splash page: a picture or a video at the top, and a button to sign up with an email address. An email address was worth a lot. It could be asked to volunteer, and asked to give.
The staff knew which video they wanted at the top. They “heavily favored” one called “Sam’s Video,” Siroker wrote. “Had we not run this experiment, we would have very likely used that video on the splash page.”
The experiment started in December 2007. It tested two parts of the page at once: six pieces of media, three images and three videos, and four different buttons. That made 24 combinations. Every visitor to the splash page was shown one of them at random, and the campaign tracked whether that person signed up. A total of 310,382 visitors went through, about 13,000 for each version.
“Everyone on the campaign loved the videos,” Siroker wrote. “All the videos ended up doing worse than all the images.” In a reply under his post, he gave the numbers: the three videos had sign-up rates of 5.95, 6.29 and 7.76 percent, the three images 8.54, 8.87 and 9.66 percent.
The best combination was a family photo with a button that said “Learn More.” It signed up 11.6 percent of visitors. The original page had signed up 8.26 percent, so the winner was 40.6 percent better.
Siroker then did some arithmetic of his own. If the improvement had held roughly steady for the rest of the campaign, he wrote, the page would have brought in about 2,880,000 extra email addresses. At the campaign’s usual rates, that came to 288,000 more volunteers and, at an average of $21 donated per address, an extra $60 million. It was an estimate built on that assumption, not a count.
The test also changed what came next. “Because this first experiment proved to be so effective,” Siroker wrote, “we continued to run dozens of experiments across the entire website throughout the campaign.” He went on to co-found Optimizely, a company that sells software for running tests like this one.