Anti-Vanity Metrics: The Numbers That Actually Tell You Something

A metric turns "vanity" the moment it can climb while the product actually gets worse — total signups, raw pageviews, or DAU inflated by ad spend, none reflecting real value delivered. An actionable metric is the opposite: tied to one decision, it moves when user behavior changes and tells you what to build or fix next.

Quick answer: Vanity metrics (DAU, downloads, followers) can rise even as the product fails users. Actionable metrics — retention curves, activation rate, a North Star Metric, and revenue-linked cohort data — move only when real behavior changes, so they actually tell you what to fix.

What Turns a Metric Into "Vanity" vs. "Actionable"?

A metric is actionable when it meets three tests: it's causal (you can trace what moved it), it's consistent across segments and time (not a fluke of one channel), and it's connected to a decision you'd actually make differently depending on its value. Fail any of the three, and you have a number that's fun to report but useless to act on.

Eric Ries drew this line explicitly in The Lean Startup, coining "vanity metrics" for numbers that "put the best face on a situation" without telling founders whether they should pivot or persevere. His counterexample was actionable metrics — ones tied to a specific, repeatable experiment, where a change in the number implies a change in what you'd do next.

The test is less about the metric's name and more about its behavior. Total registered users is nearly always vanity — it only goes up, regardless of whether new users stick around. Day-7 retention by signup cohort is nearly always actionable — it can go up or down, and a drop points you straight at a specific product change to investigate.

If you're building out a broader measurement practice rather than picking single metrics, our guide to product analytics for data-driven decisions covers how to structure the full analytics stack this table is drawn from — instrumentation, definitions, and review cadence, not just the metric names.

The Vanity Metrics PMs Keep Reaching For

Vanity metrics persist because they're easy to get, they almost always go up, and they look good in a board deck — none of which is a coincidence. They're often the only numbers a team has instrumented, so they get used as proxies for questions they were never designed to answer.

The most common offenders show up in nearly every product review deck:

1. Total downloads or signups. A cumulative counter that can never decrease, regardless of whether a single one of those users is still active.
2. Raw DAU/MAU as a headline. Useful as an input to other math, nearly meaningless alone — a spike from a push notification blast looks identical to a spike from genuine product love.
3. Pageviews or session count. More sessions can mean more engagement, or it can mean users can't find what they need and keep bouncing back to search.
4. Social followers or app store ratings. Easy to buy, easy to game, and only weakly correlated with retained, paying usage.
5. Feature "usage" counted as a click. A click registers intent, not completion — and definitely not the outcome the feature was built to produce.

None of these are useless in isolation — they're useful as inputs to a calculation, just not as outputs you report as success. The failure mode is reporting them as if they were decisions rather than raw material. A related failure: measuring against the wrong unit of analysis entirely, tracking account-level activity when the real job to be done happens at the level of an individual user's task — a mismatch our Jobs to Be Done guide walks through in more depth.

Sean Ellis, who coined "growth hacking" and built the widely used product-market-fit survey, has made a similar point about growth teams: a number going up isn't evidence of fit if it can't survive being broken down by cohort and channel.

Why They Survive Leadership Review

Vanity metrics rarely die from a lack of scrutiny — they die from a lack of alternatives. A leadership team asks "how are we doing," and if the only instrumented answer is total signups, that's the number that goes in the deck, true believers or not.

The fix isn't banning the number outright; cumulative signups is still a legitimate input to a CAC calculation. The fix is refusing to let it stand alone as the headline — pairing it, every time, with the cohort-level number that actually says whether those signups are turning into retained, valuable users.

The Metrics That Actually Change What You Build

Actionable metrics share a structure: they're measured per cohort, tracked over a meaningful time horizon, and directly attached to a specific product or business decision. Four categories cover most of what a product team actually needs.

Retention curves, not retention snapshots. A single "30-day retention" number hides whether the curve flattens (a sign of durable product-market fit) or keeps decaying toward zero (a sign it hasn't been found yet). SaaS metrics practitioner David Skok has argued that the shape of the retention curve — not any single point on it — is the clearest early signal of whether a business model actually works. Our cohort analysis guide for PMs covers how to build and read these curves without a data team.

Activation rate and time-to-value. This measures the percentage of new users who reach a defined "aha" moment within a set window — not signup, not login, but the specific action correlated with long-term retention. It directly tells you whether onboarding is the bottleneck, which no top-of-funnel number can.

A single North Star Metric with guardrails. Amplitude's North Star framework — and the broader growth-team practice it formalized — argues for one metric that captures the value a product delivers to customers, paired with a small set of guardrail metrics so optimizing the star doesn't quietly wreck something else (support load, churn, revenue quality). Choosing the wrong one is a common and costly mistake; see our guide on choosing a North Star Metric for the selection criteria.

Revenue-linked cohort metrics. Trial-to-paid conversion, expansion revenue, and LTV:CAC by acquisition channel tell you which growth is actually profitable, versus which growth is just diluting your unit economics with users who were never going to pay.

A number moving in the right direction only counts as evidence if you can also show it moved because of something you changed — which is where a properly designed experiment, not a dashboard glance, earns its keep. Our guide to experiment analysis and PM statistics covers the difference between a metric that moved and a metric that moved significantly.

Building a Metrics Stack That Survives a Board Meeting

A defensible metrics stack has three layers, not one hero number: a North Star Metric that represents customer value, a small set of input metrics that causally drive it, and guardrail metrics that catch damage the star can't see. Report all three together, every time, or the star metric turns into the next vanity metric by default.

Google's HEART framework — developed by Kerry Rodden and colleagues for measuring UX at scale — offers a useful decomposition when a single North Star feels too coarse: Happiness, Engagement, Adoption, Retention, and Task success. Each maps to a different failure mode a lone headline number would hide.

1. Happiness — direct signal (NPS, CSAT) tied to a specific interaction, not a quarterly survey in a vacuum.
2. Engagement — depth and frequency of use, always segmented by cohort so a launch's initial spike doesn't get mistaken for lasting behavior change.
3. Adoption — the rate at which new users or new features are picked up, distinct from raw counts.
4. Retention — the curve, not the snapshot, as covered above.
5. Task success — whether users actually complete the job they came to do, which is often the single most under-instrumented layer.

That last layer matters because most metrics stacks are built around product structure (features, screens, releases) rather than the customer's structure (the job they're trying to get done and the journey they take to do it). Mapping metrics to the actual path a user walks — not just the funnel a dashboard happens to track — is exactly the exercise in our customer journey guide, and it's usually where teams discover a whole stage they've never instrumented at all.

Guardrail discipline in practice: if your North Star is "weekly active projects created," a guardrail on support-ticket volume or churn keeps you from hitting the star by making the product more addictive-but-worse — a real risk with engagement-style North Stars.

Review Cadence by Layer

Each layer of the stack decays at a different speed, so reviewing all three on the same calendar is itself a quiet source of vanity-metric drift — a guardrail checked only quarterly can slip for months before anyone notices.

The Real Root Cause: Every Vanity Metric Started as an Unlogged Assumption

Most vanity-metric mistakes don't start with bad math — they start with an assumption nobody wrote down. A team assumes "more signups means more value," ships a growth campaign that hits that number, and only discovers months later that the assumption was never actually tested. The metric wasn't wrong; the unexamined belief behind choosing it was.

Teresa Torres, in her work on continuous discovery, makes assumption-mapping a first-class habit rather than an afterthought: before you trust a metric, name the assumption that makes it meaningful, and test the riskiest assumption first. Applied to analytics specifically, that means writing down, at the moment you pick a metric, exactly what you believe has to be true for that number to mean what you think it means.

In practice, that discipline breaks down for a mundane reason: the assumption gets formed in a hallway conversation, a Slack thread, or a half-formed thought during a dashboard review — and by the time the quarterly retro rolls around, nobody remembers it was ever a hypothesis rather than a fact. The vanity metric survives because the assumption that should have challenged it was never captured anywhere durable.

Consider a familiar sequence: a PM glances at a DAU chart mid-week, notices it's up, and quietly assumes it's because of last sprint's onboarding tweak. Nobody writes that belief down. Two months later, the same PM presents the DAU trend as evidence the onboarding change worked — except the real driver turned out to be a seasonal spike unrelated to any release. The metric didn't lie; the untested, unrecorded assumption about why it moved did.

This is the case for treating assumption-capture as its own discipline, not a footnote to analytics work:

• Log the assumption when you form it, not when you write the retro deck — memory reconstructs assumptions to fit whatever the data later showed, which defeats the point of having logged one at all.
• Timestamp it, so you can later check whether the metric moved before or after you believed the thing that justified watching it.
• Make it revisitable, so a metric review can pull up the original hypothesis instead of a team's current, possibly revisionist, memory of what they expected.

Key Takeaways

• A metric is vanity if it can rise while the product gets worse — total signups and raw DAU are the two most common offenders in board decks.
• An actionable metric passes three tests: it's causal, consistent across segments, and tied to a decision you'd make differently based on its value.
• Retention curves beat retention snapshots — the shape of the curve over time reveals product-market fit far more reliably than any single percentage.
• A North Star Metric needs guardrails, or optimizing it can quietly damage churn, support load, or revenue quality without anyone noticing until later.
• The HEART framework (Happiness, Engagement, Adoption, Retention, Task success) catches failure modes a single headline metric hides, especially task success.
• Most vanity-metric mistakes trace back to an assumption that was never written down — the fix is capturing it the moment it forms, not reconstructing it after the fact.
• Instrument around the customer's journey and job, not just the product's funnel — the biggest measurement gaps usually sit in stages nobody mapped.