Define success before you define the build: every spec should carry a primary outcome metric, a short list of guardrail metrics, and a plain-language measurement plan before an engineer ever sizes a ticket. Metrics chosen after launch get shaped to flatter whatever shipped, or they quietly disappear once no one's watching. Pick the metric while the spec still has nothing to protect, and it does its job: telling you, honestly, whether the feature worked.
Quick Answer: Write a
Success Metricssection into the spec before eng estimates it — one primary outcome metric, 2-4 guardrail metrics, and a named measurement method with an owner and a check-in date. Grade acceptance criteria and the outcome metric at the same ready-to-ship gate, not in two separate conversations months apart.
Why Metrics Defined After Launch Get Gamed or Forgotten
A metric chosen after a feature ships already knows the outcome, so it tends to describe what happened rather than judge whether it was good. That's Goodhart's Law working in reverse: a metric invented after the fact is just as compromised as one gamed after the fact, because in both cases the number is chosen to fit a result rather than to test one.
Economist Charles Goodhart's original observation, as commonly paraphrased in management literature: "When a measure becomes a target, it ceases to be a good measure." A metric chosen once the outcome is already known is that same failure running in reverse.
Three things reliably go wrong when metric selection waits for the post-launch retro:
- Confirmation metric-picking. Whoever runs the retro reaches for whichever number already looks good — clicks if clicks are up, tickets closed if tickets are down — instead of the number that mattered when the feature was scoped.
- No named owner, so no follow-up. A metric floated in a meeting with no owner and no calendar date attached gets mentioned once and never checked again.
- No baseline to compare against. Without a pre-launch number, "engagement went up" is unfalsifiable — up from what, measured how, over what window?
The pattern shows up in industry research on feature usage more broadly, too. Widely cited studies on shipped-feature usage — including Jim Johnson's Standish Group presentations, repeated for two decades across agile and lean-product literature (Eric Ries's The Lean Startup among them) — put the share of features that go rarely or never used at roughly half. A feature nobody defined a success metric for is a feature nobody can later prove earned its build cost, which is a large part of why that number stays stubbornly high.
Product leader Marty Cagan, writing for the Silicon Valley Product Group, frames this as the core failure of output-driven roadmaps: teams that measure "did we ship it" instead of "did it work" have no mechanism to distinguish a good bet from a bad one, so bad bets keep recurring. A spec's living-document status doesn't help if the outcome section is the one part everyone quietly agrees to revisit "later."
The table below compares what changes when metric selection happens at kickoff versus after launch — the same metric, chosen at two different moments, behaves like two different tools:
| Dimension | Metric chosen before the spec | Metric chosen after launch |
|---|---|---|
| Who's blind to the result when picking it | The team — outcome isn't known yet | No one — the result already exists |
| Influences the design | Yes — shapes scope and acceptance criteria | No — feature is already built |
| Gaming risk | Lower — target set before the number exists | High — easy to pick a flattering read |
| Accountability | Built into the ready-to-ship gate | Optional, easy to skip |
| Typical fate | Reviewed on a set date, by a named owner | Debated once in a meeting, then dropped |
Timing, not cleverness, is what determines whether a metric can be gamed. The fix isn't a smarter metric — it's an earlier one, which is really what it means to define success in the spec rather than in the retro.
The Three-Part Structure: Primary Outcome, Guardrails, and How You'll Measure
A success-metrics section needs exactly three components to be gradable: one primary outcome metric that defines what winning looks like, two to four guardrail metrics that catch damage the primary metric can't see, and a measurement plan naming the data source, the exact query or event, and a check-in date. Miss any one of the three and the section is decoration, not a spec.
Primary outcome metric
This is a single number, not a bundle of good intentions. It should read like a bet, not a wish: metric name, current baseline, target, and timeframe, in one sentence a stakeholder could repeat back. Following something close to a North Star Metric model — an idea popularized in growth circles by Sean Ellis and later formalized by analytics firms like Amplitude — the metric should represent real customer value, not a vanity number that moves without anyone actually benefiting.
Two questions filter a strong candidate from a weak one:
- Does moving this number require the feature to actually work, or could it move from a redesign, a marketing push, or seasonality alone?
- Would the team accept losing the bet if the number doesn't move — or is there already an excuse waiting?
If the honest answer to either is uncomfortable, the metric isn't specific enough yet.
Guardrail metrics
A single metric optimized in isolation invites exactly the gaming Goodhart warned about — improving the one number being watched at the expense of everything that isn't. Andy Grove built this insight into Intel's original OKR practice, as John Doerr recounts in Measure What Matters: every objective was paired with a "health metric" specifically to catch the objective being hit in a way that quietly broke something else.
The same logic underpins DORA research (Nicole Forsgren, Jez Humble, and Gene Kim's Accelerate), which never reports deployment frequency alone — it's always paired with change failure rate as a guardrail, because deploy speed without a failure-rate check just measures how fast a team can break production. A spec's guardrails should follow the same pairing instinct:
- A quality guardrail — error rate, support-ticket volume, or churn in the touched flow.
- A fairness or equity guardrail — does the change help one segment while quietly hurting another?
- A cost or performance guardrail — latency, infra spend, or support load the change might add.
Product thinker Itamar Gilad, whose GIST framework treats guardrails as a first-class planning input rather than an afterthought, argues teams should set guardrail thresholds before launch specifically so a borderline result has a pre-agreed answer instead of a debate.
How you'll measure
The measurement plan is the least glamorous part of the section, and the one most often skipped — which is exactly why skipping it is where things quietly break. It names four things: the data source (event, table, survey), the exact definition (not "activation," but "second key action within 7 days of signup"), the owner, and the check-in date.
Here's the difference between a field that sounds like a plan and one an engineer or analyst could actually execute against, using a hypothetical bulk-import spec:
| Spec field | Weak version | Gradable version |
|---|---|---|
| Primary metric | "Improve activation" | activation rate (2nd key action / 7 days) from 34% baseline to 42% by day 60 |
| Guardrail 1 | "Don't break anything" | Support tickets tagged import-error stay flat or fall, weekly |
| Guardrail 2 | "Keep it fast" | p95 import completion time stays under 90 seconds |
| Measurement source | "We'll check analytics" | import_completed event in the product analytics warehouse, dashboard owned by PM |
| Check-in date | "Later" | Day 30 and day 60 after full rollout, on the PM's calendar |
Framed this way, the section reads less like a promise and more like a contract — one that a PR-style spec diff review can hold the team to exactly as strictly as it holds scope or copy.
Connect Metrics to Acceptance Criteria and the Ready-to-Ship Gate
Acceptance criteria describe what the feature does; the success-metrics section describes why doing it was worth the eng cost — a spec is only complete when both agree on the same definition of "done." At the ready-to-ship gate, the metrics section deserves the same scrutiny as code review: is the baseline captured, is instrumentation shipping in the same release, and does a named owner have a date to check the number?
In practice, this means treating instrumentation as an acceptance criterion, not a nice-to-have:
- "Event X fires with property Y" belongs in acceptance criteria alongside "button does Z." If the QA test cases derived from the spec don't check the tracking event, the metric will be unmeasurable on day one regardless of how well the feature works.
- Baseline capture is a pre-launch task, not a post-launch one. If the current number isn't pulled before the change ships, there's no "before" to compare the "after" against.
- The ready-to-ship gate should block on a missing metrics section, the same way it blocks on missing edge-case handling. A feature with no named owner for its own outcome isn't actually ready — it's just built.
This is also where a spec's section-level maturity matters: a success-metrics section still sitting in draft shouldn't be able to hide behind acceptance criteria that look polished. If scope is marked "ready" while metrics stay a placeholder, the gate is only checking half the spec. Treat the two as maturing together, or the readiness signal is misleading.
Amazon's internal "working backwards" process — documented publicly in Colin Bryar and Bill Carr's Working Backwards — is a useful reference point: the mock press release and FAQ that precede any build are explicitly required to state how the team will know the effort succeeded, before a single line of code is estimated. The mechanism differs from a PRD's success-metrics section, but the discipline is identical: name the outcome while it's still cheap to change your mind about the build.
A Worked Example: How Naming the Metric Early Changed the Design
Naming the metric before the design is finished routinely changes the design itself, because the metric reveals which failure mode actually matters. Consider a pattern that recurs across many B2B tools building a bulk CSV import feature — not a single case study, but a composite of a decision teams run into constantly, useful precisely because the mechanism generalizes.
Picture two versions of the same spec kickoff.
Version A — metric deferred. The spec scopes "let users upload a CSV of records instead of adding them one at a time." Acceptance criteria cover the happy path: valid file uploads, records appear. The team builds a straightforward upload-and-process endpoint, ships it, and only after launch does someone ask how to tell if it worked — by then the natural answer is "uploads completed," a number that says nothing about whether the import actually helped anyone.
Version B — metric named first. Before scoping, the PM writes the primary outcome metric: reduction in import-error support tickets within 30 days of a customer's first import, not upload completions. Naming that metric surfaces a question the happy-path spec never asked: what happens when the file is almost right — a malformed date column, a duplicate ID? That single question changes the acceptance criteria to require inline validation and a preview screen before commit, not just a background job that dumps a failure log nobody reads.
The mechanism is simple: an outcome metric forces the spec to account for the failure path, because the failure path is what the metric is actually watching. A metric like "uploads completed" only ever asks about the happy path — it structurally cannot catch the case that generates support load. This is the same instinct behind mapping a customer journey's emotional friction points before scoping a fix: the moment of highest frustration, not the moment of technical completion, is usually where the real design requirement lives. It's also why grounding the metric in a customer's actual job to be done, rather than a proxy for effort, tends to catch this earlier.
Make Success Metrics a Living Part of the Spec, Not a Postscript
A success-metrics section stays honest only if it's treated as a first-class, adaptive part of the spec — present at kickoff alongside scope and acceptance criteria, revised through the same review as everything else, not an appendix bolted on when a stakeholder finally asks "so did it work?" Specs that isolate metrics in a separate doc or an end-of-quarter deck make it easy to quietly let the metric slide when the number disappoints.
This is the specific gap Prodinja's Spec Studio is designed to close. Its adaptive sections let a Success Metrics block become a standard, versioned part of the living PRD — reviewed with the same PR-style diffs as scope changes, checked at the same readiness gate as acceptance criteria, and handed off with the rest of the document rather than tracked in a separate slide deck. The point isn't a smarter metric; it's making it structurally harder for the outcome to be defined after the fact, because the section already exists before there's anything left to hide.
Key Takeaways
- Pick the metric before the outcome exists. A metric chosen after launch is chosen with the result already known, which is why it gets gamed or quietly dropped.
- Structure the section in three parts: one primary outcome metric, 2-4 guardrail metrics, and a plain-language measurement plan with an owner and a date.
- Guardrails exist to catch what the primary metric can't see — quality, fairness, and cost effects the main number won't reflect on its own.
- Instrumentation is an acceptance criterion, not an afterthought — if the tracking event isn't in QA's test cases, the metric will be unmeasurable at launch.
- The ready-to-ship gate should block on an undefined metric exactly the way it blocks on missing edge cases — a feature with no named outcome owner isn't done, it's just built.
- Naming the metric early routinely changes the design, because it forces the spec to account for the failure path the metric is actually watching.
- Goodhart's Law cuts both ways: a metric invented to fit an outcome that already happened is just as compromised as one gamed after the fact.
Frequently Asked Questions
What's the difference between a primary outcome metric and a guardrail metric?
The primary outcome metric defines what winning looks like — the one number the feature exists to move. Guardrail metrics don't define success; they define acceptable cost, catching quality, fairness, or performance damage the primary metric would never reveal on its own.
How many guardrail metrics should a spec include?
Two to four is a workable range for most specs. Fewer than two leaves obvious blind spots, like quality or cost; more than four dilutes attention and makes the ready-to-ship review unwieldy — a fifth guardrail is often a sign the primary metric was scoped too broadly.
Should engineering estimate the cost of instrumentation, not just the feature?
Yes — instrumentation is part of the build, not a follow-up task. If the event, log, or query that will produce the metric isn't estimated and scheduled alongside the feature, it routinely ships late or not at all, and the metrics section becomes unmeasurable on day one.
What if the primary metric takes months to move — how do you stay accountable sooner?
Pair the long-horizon outcome metric with a short-horizon leading indicator believed to predict it — an activation step, an early-retention curve — and check that one at 30 days while the outcome metric matures over a longer window. The leading indicator doesn't replace the outcome metric; it just gives the team an earlier honest signal.
Who should own the success-metrics section — PM, data team, or engineering?
The PM typically owns the primary outcome metric and the overall narrative, since it's a product bet; a data or analytics partner usually owns the measurement definition and instrumentation accuracy; engineering owns building the instrumentation itself. All three should sign off before the spec clears its ready-to-ship gate.