On December 6, 2012, Vic Gundotra, a senior vice president at Google, posted on the company’s official blog to launch new Google+ features. His second paragraph began with a claim: “Today Google+ is the fastest-growing network thingy ever.”
Then he gave the size of it, three times in one sentence. More than 500 million people had upgraded, he wrote. 235 million were active across Google. And counting only activity in the stream itself, 135 million were “active in just the stream.”
The three figures counted different things: people who had upgraded, people active somewhere across Google, and people active in the stream alone. Each could be true by its own definition. Each answered a different question.
None of them said how long anyone stayed.
Feeling good, knowing nothing
A number that makes you feel good isn’t necessarily one that tells you what to do next.
Eric Ries, a co-founder and former chief technology officer of IMVU, a 3D chat company, put a name to this kind of number in a 2009 guest post on Tim Ferriss’s blog. Most of the data in off-the-shelf analytics packages, he wrote, are what he called vanity metrics. “They might make you feel good, but they don’t offer clear guidance for what to do.” His example was the total number of hits to your website. Say you have 10,000. Now what?
The usual culprit is a running total. Total sign-ups or total accounts, a count of everyone who ever arrived, can only go up, and it can go up for reasons that have nothing to do with whether the product works. A launch can push it. So can a change to a sign-up form. A gym’s count of everyone who ever signed up climbs every January and never falls. The door swipes tell you whether anyone comes to lift.
Google’s totals may have had help of that kind. On January 20, 2012, the day after Larry Page announced 90 million Google+ users, TechCrunch noticed that creating a new Google account now meant creating a Google+ presence too. The writer said it “smells of an attempt to boost G+ numbers.” A few weeks later, comScore figures reported by The Wall Street Journal, as Poynter summarized them, put visitors’ time on Google+ at about three minutes a month on computers, against six to seven hours on Facebook. Those figures left out phones.
What Ries suggested instead were numbers that answer a question: split tests, where half your customers see a change and half don’t, and per-customer and cohort figures, which follow each week’s new arrivals to see what they go on to do. If a cohort’s numbers hold steady from week to week, nothing important has changed. If one suddenly moves, he wrote, you get “a rapid signal to investigate.”
A label for the use, not the number
No number is vain on its own. Total accounts matter to the engineer sizing the servers. The same total is vanity when it sits at the top of a deck as proof that people like the product, because it would have risen either way.
Ries’s standard was whether a number helps you make a decision. One way to apply it: ask what you would do differently if the number came out lower. If the honest answer is nothing, it is decoration. Cohort numbers aren’t immune, either. A cohort of forty people can swing by chance, and a sharp move is, in Ries’s words, a signal to investigate, not a verdict.
The launch that lifted sign-ups 40 percent
Imagine you run a small app for booking dog walkers, and this month’s investor update has a lovely chart: sign-ups up 40 percent after a referral campaign. Before you send it, you split the sign-ups by the week they joined and look at one thing: how many booked a second walk within 30 days.
For the weeks before the campaign, it’s about one in four. For the campaign weeks, it’s about one in ten. You haven’t learned why yet. The referral bonus may have brought people who wanted the bonus, or people in towns with no walkers. But you now have a question worth a week of work, and an update that says what you don’t know.
Number two on the leaderboard
AI companies compete on leaderboards, and the rank becomes the headline. When Meta launched its Llama 4 models on April 5, 2025, its announcement said an “experimental chat version” of the Maverick model scored an Elo of 1417 on LMArena, a site where people compare two models’ answers and vote for the better one. That put it second.
Researchers soon noticed that the version on LMArena wasn’t the one developers could download. Meta’s own materials said it was “optimized for conversationality.” LMArena posted that Meta “should have made it clearer” that it was “a customized model to optimize for human preference,” and changed its rules. Meta said it experiments with “all types of custom variants.” Once LMArena scored the released model, TechCrunch reported on April 11, it ranked below several older models, including OpenAI’s GPT-4o.
The ranking was real for the model that earned it. For anyone choosing a model to build on, it measured something they couldn’t use. The check that matters is narrower and duller: take twenty tasks from your own work, write down what a good answer looks like before you start, and run the exact model version you’ll deploy on each task several times. Answers vary from run to run, so look at the pattern across runs, and read the misses against your own standard.
In October 2018, Google announced it would shut down Google+ for consumers. This time the number it published was about how long people stayed. “90 percent of Google+ user sessions are less than five seconds.”