Late on January 31, 2017, at around 11 p.m. his time, an engineer at GitLab had said he was going to sign off. Then the database replication broke, and he stayed.
GitLab.com, where developers keep their code and track their work, ran on two database servers. The primary, db1, did all the work. The secondary, db2, kept a copy in case db1 failed. Earlier that evening a surge of load, which GitLab suspected was spam, had left db2 so far behind that it stopped copying altogether. The fix was to wipe db2 and copy everything over fresh.
The copy would not start. The team changed settings, restarted the database, ran a debugging tool. The copy just sat there, printing nothing. The engineer guessed that leftover files in db2’s data folder were in the way, and ran the command to delete the folder.
A second or two later he noticed where he had run it: db1, the primary. He stopped it at 11:27 p.m. UTC. Of around 300 GB, about 4.5 GB was left.
(According to GitLab’s postmortem, the copy had never been stuck. The tool waits silently for the primary to start sending data, and GitLab’s runbooks did not make that clear.)
This is the moment a backup is supposed to make the story boring. GitLab had several. As the team searched, they wrote what they found into a Google document that GitLab later made public.
The daily database dumps had been running an old export tool that didn’t work with the newer database, and producing nothing. The job did send failure emails, but they were not signed the way the receiving server required, so they were rejected and GitLab never learned the backups were failing. The Amazon S3 bucket where the dumps should have been was empty. Disk snapshots in Azure were switched on for the file servers but not for the databases. The replica was db2, which the team had just wiped themselves. By the count in those live notes, five backup and replication methods were in place, and none of them was working reliably or had even been set up.
What saved them was a coincidence. About six hours before the deletion, the same engineer had taken a snapshot of the database by hand, for his load-balancing work. It sat in a test environment that ran on cheap, slow disks, and copying it back took around 18 hours. GitLab streamed the recovery on YouTube. By its count about 5,000 people watched at the peak, and for several hours it was the number two live stream on the site.
GitLab.com came back at 6:14 p.m. UTC on February 1. About six hours of database changes were gone, by GitLab’s estimate roughly 5,000 projects, 5,000 comments and 700 new user accounts. Code repositories were not affected.
Ten days later GitLab published a postmortem. It said its main focus would be making mistakes easy to recover from, not stopping engineers from running certain commands. Among the fixes: a clearer prompt on every server, showing which machine you are typing into.
The last of its whys asked why the backups had never been tested on a regular basis. The answer: “Because there was no ownership, as a result nobody was responsible for testing this procedure.”