At 04:09 UTC on Friday, July 19, 2024, CrowdStrike pushed out a routine update for its Falcon security software on Windows. It wasn’t a new version of the program. It was a small configuration file meant to help the software watch for new attack techniques. In the company’s own words, it was “not code or a kernel driver.”
Computers that received it crashed to the blue screen. At 05:27, seventy-eight minutes later, CrowdStrike pulled the file. Machines that came online after that were fine. Some of the ones that already had it kept crashing before they could download the fixed version.
Microsoft later estimated that 8.5 million Windows devices were affected, less than one percent of all Windows machines. The trouble was which one percent. Microsoft pointed out that CrowdStrike is used by enterprises that run many critical services. Flights were grounded in Europe and the US. In Britain, a minister told Parliament that large parts of local healthcare had lost access to test results and appointment information, mostly at GP surgeries. As the Guardian put it, the software meant to protect vital systems “ended up taking them down.”
The fix CrowdStrike published within hours was manual. Boot the machine into Safe Mode, go to a CrowdStrike folder deep inside Windows, find the file whose name starts with C-00000291, delete it, restart. If the disk was encrypted, you might need a recovery key first. At first, it had to be done on each affected machine.
CrowdStrike’s preliminary review explains why the file had gone out with confidence. The detection type behind it had passed a stress test in March. Four earlier updates of the same kind had worked fine in customers’ machines. And there was “trust in the checks performed in the Content Validator.”
The flaw was hiding inside that record of success.
Back in February, CrowdStrike had added a new kind of detection to its sensor. Its definition said it would receive 21 pieces of information. The sensor code actually handed it 20. The gap went unnoticed for months, because the tests, and those four live updates in March and April, all put a wildcard in the 21st slot. Match anything. The sensor never had to look there.
On July 19, two new updates went out, and one of them asked about the 21st slot for real. The Content Validator, the check that runs on content before it is published, compared it with the definition, which said 21, and approved it. On customers’ machines, the sensor reached for a 21st value that wasn’t there, read memory past the end of its list, and took Windows down with it.
It also went out without a staged rollout, the practice of sending an update to a small group of machines first and watching what happens before going wider. Seventy-eight minutes was enough to reach millions.
The company’s root cause analysis, published 18 days later, listed six findings. The last one was a single sentence: “Each Template Instance should be deployed in a staged rollout.”