At 07:46 UTC on Tuesday, April 5, 2022, a support ticket reached Atlassian from a customer whose site had just been deleted. Atlassian’s own monitoring hadn’t flagged anything. The sites were being deleted through a standard workflow, so to the monitoring it looked like normal work.
A script had started eight minutes earlier, at 07:38. It was working down a list, one site at a time. By the time it finished at 08:01, it had deleted 883 sites belonging to 775 customers. Their Jira, Confluence and other Atlassian products went dark.
The job itself was housekeeping. In 2021, Atlassian had bought an app called Insight and built it into Jira Service Management, so the old standalone version had to be removed from the customer sites that still had it. One team asked another to do it with an existing script. Between the two teams, the list changed meaning. Instead of the ID of the app on each site, the list that was handed over held the ID of the whole site.
No warning came up. The deletion API took a site ID or an app ID and assumed whichever it got was right. The script also had two modes, one that marks data for deletion so it can be recovered and one that deletes it for good. According to Atlassian’s CTO, Sri Viswanath, it was run “with the wrong execution mode and the wrong list of IDs.”
The checks were there. The script had been peer reviewed, but the review looked at which endpoint it called and how, not at what the IDs pointed to. It had been tested in staging, where those IDs didn’t exist. An earlier production run had removed the old app from 30 sites without trouble, because its IDs were gathered before the mix-up.
Early on April 6, Atlassian told affected customers in their support tickets that a maintenance script was to blame and that it expected minimal data loss. On the data, by its own later account, it was right: no customer lost more than five minutes of it. It kept 30 days of backups and regularly restored customers who had deleted their own data. It could even restore every customer at once into a brand new environment.
Getting the data back was the hard part. What Atlassian had never built was a way to put a few hundred customers back into the live system, where each data store also held data for other customers who were still working. Engineers had to pull each site’s pieces out of the backups by hand.
The first method took about 70 steps per site. The first customers got their sites back on April 8, but at that pace, by Atlassian’s estimate, restoring everyone would have taken three weeks. On April 9 it proposed a faster method and changed course in the middle of the incident. The last sites were restored on April 18. For some customers, the outage lasted up to 14 days.
Meanwhile, most affected customers couldn’t even reach support through the normal form. The script had deleted their contact details along with their sites, including who each site’s administrators were. That form required a valid site address, and those addresses no longer existed. Atlassian posted on its status page within hours, but what its review calls its first broad external message, a tweet, went out on April 7, about 41 hours after the script ran.
When Atlassian first sized the damage, it estimated about 400 customers. Its final review counted 775.