Home › Stories
Spotify · 2016–2021

The Tenth Pull Request

Spotify was hiring fast, but by its own account the extra people were slowing it down. A new engineer needed more than 60 days to get a tenth piece of code merged. Ordering every team onto one system wasn't an option, so the platform team set out to build something they would choose.

◷ 2 min read

In 2016 Spotify was hiring as fast as it could, and by the company’s own account the new people were slowing it down. Its Platform team, which builds the internal tools other engineers use, tracked one number for onboarding: how many days it took a new engineer to get a tenth pull request, a tenth piece of code, merged. The answer was more than 60. Years later, on a company podcast, Spotify’s head of research and development asked Pia Nilsson, an engineering director who had joined that year, whether the number had seemed acceptable. “Having spent 15 years as an engineer at other companies,” Nilsson said, “it was like a cold shower.”

Spotify was famous for small, autonomous teams that each built what they needed. The Platform team saw the company’s developers as its customers, so it asked them what was in the way. Two answers came back. People were interrupted constantly: “New joiners had to tap someone on the shoulder because very seldom was there any documentation.” And people couldn’t find anything. By the company’s account there could be 15 versions of the same service, each built for a slightly different team, and a team that needed one would often just build another. Engineers had a name for the only way to learn how anything worked, which was to ask a colleague: “rumour-driven development.”

The obvious fix was to centralize. Nilsson’s team ruled it out. “We basically knew we couldn’t build a centralized solution,” Nilsson said. “It would never work. No one would use it.” Tyson Singer, a Spotify engineering vice president, put it more bluntly: “if we go out and we tell people to do X, they just shrug, and they do wherever they want. So we really do have to sell to them.”

So the team built Backstage: one place where anyone could find every service, see who owned it and read its documentation. Teams kept owning their own pieces and plugged them in. Spotify later reported that 280 engineering teams used it to manage more than 2,000 backend services, and that 80% of contributions came from engineers outside the Backstage team. Spotify released an open-source version in March 2020.

The onboarding number, by Spotify’s count, went from more than 60 days to a stable 20. The figures come from Spotify’s own blog and podcast, and over the same years new engineers also went through tutorials and a week-long bootcamp, so the drop can’t be pinned on one tool alone. In 2021 Spotify said 80% of its internal users were satisfied with Backstage.

What this story shows

Four Fundamental Team TypesBy Spotify's account, its autonomous teams kept rebuilding near-copies of the same services, and new engineers needed more than 60 days to merge their tenth change. Its Platform team treated the other engineers as customers and built Backstage, one place to find every service and its owner. Spotify says onboarding then fell to 20 days.

Sources

  1. Spotify Engineering, A Product Story: The Lessons of Backstage and Spotify's Autonomous Culture, May 2021
  2. Spotify Engineering, How Backstage Made Our Developers More Effective, and How It Can Help Yours, Too, September 2021
  3. Thoughtworks Technology Podcast, Developer Effectiveness, with Pia Nilsson
  4. Spotify Engineering, How We Use Golden Paths to Solve Fragmentation in Our Software Ecosystem, August 2020