Before Microsoft, Steve Ballmer had worked at one other company. He marketed brownie mix and blueberry muffin mix “for one of the largest consumer products companies.”
In October 2002, by then Microsoft’s chief executive, he brought up that old job early in a letter to customers. Consumer-goods marketers, he wrote, work out what people want in cake mix or shampoo from “the huge volume of data” fed back by millions of daily purchases. Software had nothing like it. When a program froze or crashed, customers rarely called support. They closed the program, and the bug stayed hidden.
A small team in Microsoft’s Office group had built a way to catch those moments, and by 2002 it was in Office, Windows and most of the company’s other major products. When something went wrong, a customer could send Microsoft an error report. Ballmer was frank about the risk of software that would “phone home” like E.T. Microsoft designed a compact format, the “minidump,” so a report could go out “in a few seconds with a single mouse click.” The reports, he promised, were used “for no other purpose than to find and fix bugs.”
Engineers read the stack details, the loaded modules, the type of exception. And a shape appeared in the pile. “About 20 percent of the bugs cause 80 percent of all errors,” Ballmer wrote. Then came the part he called “stunning to me”: “one percent of bugs cause half of all errors.”
That changed the order of the work. Microsoft could now “prioritize debugging work,” Ballmer wrote, to get “the biggest improvement in customers’ experience.” With Office XP Service Pack 2, Ballmer wrote, error reporting helped eliminate more than half of all Office XP errors. During the first beta of Visual Studio .NET, it let the team log and fix 74 percent of all crashes reported.
The same data reached other companies’ bugs. Some 450 firms had looked up error reports involving their drivers, utilities and applications. Errors involving third-party firewall software, Ballmer wrote, had dropped 67 percent since the start of the year.
Seven years later, Microsoft engineers published a paper on the system, by then called Windows Error Reporting and running on about a billion computers. They described what happens to the curve over time. As a team finds and fixes the bugs users hit most often, service pack after service pack, the curve flattens out. The big holes go first, and each fix after that buys a little less than the one before.