At 04:09 UTC on July 19, 2024, CrowdStrike sent out a small configuration file for its Falcon security software. Updates of this kind, meant to help the software detect new attack techniques, had gone out several times before without trouble.
It went out in one go to customers’ Windows machines running the Falcon sensor. There was no small first group, no pause to see what happened. Machines that were online and picked it up crashed to a blue screen.
At 05:27 CrowdStrike pulled the file. Machines that came online after that were fine. By then, Microsoft later estimated, 8.5 million Windows devices had been hit. That was less than one percent of all Windows machines, but the damage was wide, Microsoft said, because of “the use of CrowdStrike by enterprises that run many critical services.” The case has the rest of the day.
CrowdStrike’s root cause analysis, published 18 days later, listed six findings. The last was one sentence long: “Each Template Instance should be deployed in a staged rollout.” Now each update goes first to a small canary group, then to successively wider rings, with a wait at every step while the company gathers data on how it is behaving. Customers can also choose where and when updates land.
Small first, then wider
A batch is whatever you finish and pass on in one go: a release, a shipment, a pull request, a day’s worth of forms. Making it smaller does two things.
It brings feedback sooner. The DORA research program, which studies how software teams deliver, puts it plainly: small batches reduce “the time it takes to get feedback on changes, making it easier to triage and remediate problems.” If the first ten machines crash, you learn that from ten machines.
It also limits the damage when something does go wrong. Amazon drew the same lesson from its 2017 S3 outage, when one mistyped command took down a subsystem that every request depended on. Its fix included speeding up work to split that subsystem into smaller pieces it calls cells, to reduce the “blast radius” of a failure and make recovery faster.
Small batches also cut waiting. Work in a big batch sits until the rest of the batch is done, so even the first item isn’t finished until the whole batch is. That is Little’s law again: at the same rate of output, the more work sitting in the system, the longer each piece stays there on average.
The invitations
Imagine someone sending out forty party invitations. The efficient-seeming way is to fold all forty, then stuff all forty envelopes, then seal them, then stamp them. The first finished invitation appears near the end of the evening. If the cards turn out not to fit the envelopes, that becomes clear at envelope one, after all forty are folded.
Done one at a time, the first invitation is ready in a minute. A problem with the envelopes shows up on the first card, before the other thirty-nine are folded to the wrong size.
Small batches aren’t free. Each one carries a fixed cost: setting up, handing over, checking. DORA notes that one reason teams work in big batches is the high fixed cost of handing work on, and that continuous delivery tries to change those economics so small batches become affordable. Another trap is to work in small pieces and then bundle them back together before release, which throws the early feedback away.
Small batches for AI work
AI coding tools push the other way. They can produce a whole feature in minutes. DORA’s guidance says that while AI “excels at generating large blocks of code quickly, large changes are difficult to review, test, and integrate safely.” It goes further: the effort of reviewing machine-written code “may be higher per line than reviewing human-written code.” Its research, it says, shows that working in small batches “amplifies the positive impact of AI adoption on product performance.”
In practice that means asking an agent for one small change at a time and checking each before the next, rather than accepting one enormous change at the end. The same goes for putting an agent into service. A new customer-reply agent, or a new set of instructions for an old one, can answer a small slice of real cases first, with someone reading what it sends, before it answers all of them.
CrowdStrike’s update was small. What went wrong was that it went out widely in one push, with no small batch to try it on first.