Late on January 31, 2017, an engineer at GitLab ran a command to clear out a database folder on the backup server. A second or two later he saw which machine he was on. It was the primary, the one that held the live data. By the time he stopped it, about 4.5 GB was left of around 300.
There should still have been a way back. “Normally in an event like this,” GitLab’s postmortem says, “one should be able to restore a database in relatively little time using a recent backup.” The engineers went looking for the backups.
The daily dumps had produced nothing for some time: they ran an export tool that didn’t work with the newer database. The job did send failure emails, but the emails were rejected by the receiving server, so nobody saw them. That was only the first backup they tried. By the count in GitLab’s own notes, five backup and replication methods were in place, and none was working reliably, or had even been set up.
Slices of cheese
The British psychologist James Reason described why this keeps happening. He first drew the layers in 1987, and the cheese was added a few years later by an Australian air-safety investigator. Dangerous systems are guarded by layers, he wrote in the BMJ in 2000. Some are engineered, like alarms and automatic shutdowns. Some are people, like pilots and surgeons. Some are procedures. In an ideal world each layer would be intact. “In reality, however, they are more like slices of Swiss cheese, having many holes.”
The holes are not fixed. They are “continually opening, shutting, and shifting their location.” A hole in one slice usually does no harm, because the next slice covers it. An accident needs the holes in many layers to “momentarily line up.”
At GitLab, each hole on its own was survivable. A broken dump job is fine if snapshots exist. Missing snapshots are fine if the dumps work. A deleted folder is fine if anything else works. On that night, nothing did.
The picture has a trap of its own. Five slices drawn on a whiteboard are not five independent protections. If several layers share the same account, the same untested process or the same assumption, one hole goes through all of them at once. Asked why disk snapshots weren’t switched on, GitLab’s engineers wrote: “We assumed our other backup procedures were sufficient.” Asked why the backups weren’t tested regularly: “Because there was no ownership.”
Two kinds of hole
Reason sorted the holes into two kinds. Active failures are the slips, lapses and mistakes of the people at the sharp end: the command typed on the wrong server. Latent conditions are what he called the “resident pathogens” of a system, laid down earlier by designers, builders, procedure writers and managers. An email check nobody tested. A backup nobody owned. They can sit quietly for months, until an active failure comes along and finds them.
“Active failures are like mosquitoes,” he wrote. “They can be swatted one by one, but they still keep coming.” The better remedy is to drain the swamps they breed in. So when something goes wrong, “the important issue is not who blundered, but how and why the defences failed.”
Imagine a waiter at a busy Friday lunch who takes an order for a child with a nut allergy and writes it on the ticket. The ticket printer in the kitchen cuts off notes after one line, so the allergy is half missing. The cook who usually reads tickets aloud is off sick. The dessert has a new garnish that nobody added to the allergen sheet. Each of these is a small thing. Together they put a plate of praline in front of the child. The waiter did everything right. Replace him, and the printer and the allergen sheet are still broken.
A slice made of training
AI systems are guarded by slices too, and in 2025 Anthropic published an account of watching several of them at work at once.
Attackers, whom the company assessed as a Chinese state-sponsored group, used its coding agent, Claude Code, against roughly thirty organizations around the world. The first slice was Claude itself, “extensively trained to avoid harmful behaviors.” The attackers got through it. They broke the attack into “small, seemingly innocent tasks,” and told Claude it worked for a legitimate cybersecurity firm doing defensive testing. The campaign succeeded in a small number of cases.
A later slice held. In mid-September 2025, Anthropic’s monitoring detected the suspicious activity. Over the following ten days it banned accounts, notified those affected and worked with the authorities. Anthropic also noticed a stroke of luck. Claude sometimes made up login details or claimed to have found secrets that were in fact public, which got in the attackers’ way. That was a lucky hole in the attack, not a slice anyone could rely on.
Anthropic’s response was the one Reason would have predicted: more and thicker slices. It expanded its detection and built better classifiers to flag malicious activity.
GitLab’s fix ran the same way. Its list of follow-up work included a new database restore process and assigning “an owner for data durability.” A backup is a slice only once someone has tried restoring from it.