Mid-November 2013, a rented room in Columbia, Maryland, lined with screens. The people trying to save HealthCare.gov were at one of their twice-daily stand-up meetings when an engineer called out from the back that a brief outage had probably been his mistake.
Mikey Dickerson, who ran the meetings, demanded a round of applause.
About six weeks earlier, HealthCare.gov, the website at the heart of President Obama’s health law, had opened to the public. On October 1, outages began within two hours. By the count of HHS’s inspector general, six people managed to submit an application and choose a plan that day. By October 17, only 3 in 10 visitors could get on at all, and according to Steven Brill’s reconstruction in TIME, the President was thinking of scrapping the site and starting over.
Dickerson was a site reliability engineer at Google. On October 18 he joined a conference call so garbled he couldn’t tell who was running it. “Finally I jumped in and asked, ‘Who am I talking to? Who is leading this call?’” It was the White House’s chief technology officer. Dickerson made plans to fly out for a few days.
The same question hung over the whole project. When the inspector general later asked CMS staff who had led it, several named positions that were unfilled. CMS had three separate war rooms.
The first outsiders on site found no dashboard showing how many people were on the site or where traffic was stuck, and spent their first night building one. Then they found a quick win: the site went to its main database for information that busy sites keep in a fast cache. On October 22, caching cut page loads from eight seconds to two. The next day the team told the White House it could have the site working for the vast majority of visitors by the end of November. “It’s just a website,” Dickerson said. “We’re not going to the moon.”
From October 24 he held a stand-up every day, weekends included, at 10:00 a.m. and 6:30 p.m. Bugs picked in the morning were supposed to be cleared by the evening. Dozens of the engineers he led worked for the contractors who had built the broken site. Dickerson posted his rules on a wall outside the control center. Rule 2 said the people who knew most about an issue should do the talking, not the ones with the highest rank. Rule 1 said:
“The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame.”
Less than three days in, the whole site crashed. A switch had failed during maintenance at a data center, and with no website to look at, the team could do little. Two days later a storage unit failed and the site went down again, through the afternoon the health secretary testified before Congress. “Those outages were totally demoralizing,” one of the engineers said.
The stand-ups kept going, twice a day, all the way to the one where an engineer owned up.
The meetings were not the whole rescue. Hardware upgrades had dramatically increased capacity, and a separate team had cut the error rate from 6 percent in October to 0.5 percent by December 1. On December 23, the original deadline for coverage starting January 1, the site took 129,000 enrollments, about five times as many as in all of October.
Dickerson had planned to stay a few days. He went back to Google on January 5.