A team that loves null results isn't lowering its standards — it's finally measuring the right thing. If roughly 70–80% of well-designed experiments show no meaningful lift, treating every one as a failure burns morale for no reason; treating it as a documented answer to a real question turns the same test into permanent, reusable knowledge.
Quick Answer: Mature experimentation programs typically see a 20–30% win rate, not 50%. The other 70–80% aren't wasted — they're valid, closed questions. Score experiments on whether they produced a decision-worthy answer, log the hypothesis and outcome with the same rigor as a winner, and losses start compounding into strategy instead of disappearing into a backlog.
Why Most Experiments Are Supposed to Fail
Most experiments test a real uncertainty, so the honest win-rate base sits around 20–30%, not 50%. Ronny Kohavi's research running Microsoft's experimentation platform found only a fraction of tested ideas — commonly cited in the 10–33% range — actually improved the target metric; the rest were flat or negative, and that ratio holds across most mature programs.
This surprises new growth teams because intuition works backward: you only greenlight ideas you already believe in, so it feels like most should win. They don't, because belief isn't evidence — that's the entire reason you're running the test instead of just shipping the idea.
Booking.com's experimentation program, described publicly by its own leaders including former head of experimentation Lukas Vermeer, has put the number even higher — roughly nine in ten tests there fail to beat the control, at a company running thousands of concurrent experiments a year.
| Source | Reported Win Rate | Program Context |
|---|---|---|
| Ronny Kohavi (Microsoft experimentation research) | Roughly 1 in 3, sometimes lower | Large-scale online controlled experiments across product teams |
| Booking.com (as its experimentation leaders have described publicly) | Roughly 1 in 10 | High-volume, mature testing program |
| Stefan Thomke, Harvard Business School | Directionally similar — most tests don't beat control | Cross-company research on experimentation culture |
A win rate near 50% is usually a red flag, not a badge of honor. It typically means the team is testing changes too timid to move a real metric, or the MDE (minimum detectable effect) is set loosely enough that noise starts looking like signal. Harvard Business School's Stefan Thomke makes a related point in his research on experimentation culture: teams that get the most value from testing are willing to test ideas with real uncertainty attached, including ones leadership privately doubts.
Three quick diagnostics catch a win-rate number that's quietly lying to you:
- Win rate above ~40% for more than two quarters — check whether tests are being called early, or the
MDEis set loose enough that anything reads as a lift. - Win rate that barely moves quarter to quarter — suspicious regularity often means the same low-risk experiment type is being repeated instead of testing riskier ideas.
- No experiments ever classified as a confirmed negative — if nothing loses cleanly, tests are probably getting killed early instead of read out honestly.
This is exactly why the growth PM role is built around finding true things quickly rather than shipping winners on demand — the job was never to guess right every time. Statistical literacy and comfort reporting a loss are part of what makes growth work distinct from adjacent roadmap-execution roles, since the sheer volume of tests run means losses are the statistical majority of the job, not an exception to manage around.
The Real Cost of Miscounting a Failed Experiment as a Bad One
A "failed" experiment and a bad experiment are different things, and conflating them is what actually kills experimentation cultures. A well-designed test that finds no effect is a resolved question; a bad test — underpowered, contaminated, or built on a vague hypothesis — is the only real failure, because it doesn't answer anything at all.
Break the outcome into three real categories instead of a binary win/lose, and the honest picture gets much clearer:
| Outcome | What Happened Statistically | What It Actually Teaches | Next Action |
|---|---|---|---|
| Confirmed negative | Treatment measurably worse than control | The core assumption behind the idea was wrong | Kill the idea, document why, don't retest without new evidence |
| True null | No statistically significant difference, adequately powered | This lever doesn't move this metric | Close the question, redirect effort elsewhere |
| Inconclusive | Underpowered, contaminated, or stopped early | Nothing — the test didn't run long or clean enough to say | Rerun with a fixed design, or discard entirely as non-evidence |
Only the third row is a genuine failure, and it's a design failure, not a results failure. This is why win rate is the wrong single metric for a growth team's health. Track a second number alongside it: learning rate — the share of closed experiments that landed in row one or two, versus wasted in row three. A team can have a mediocre win rate and a great learning rate, and that's a healthy program.
Consider a hypothetical checkout redesign test: the treatment shows a small negative movement on conversion, comfortably outside the noise band, at full statistical power. That's a confirmed negative, not a failure — the team now knows the redesign direction doesn't work for this audience, which is exactly why they ran the test before shipping the redesign to everyone.
This distinction matters most when deciding where to spend testing capacity in the first place. Because traffic and attention are finite, funnel ownership and leverage should drive which stage gets the next test slot — a confirmed null on a high-leverage step is worth more than an ambiguous "inconclusive" result on a low-traffic edge case, even though neither one is a win.
How to Avoid HARKing After a Null Result
HARKing — hypothesizing after the results are known — means presenting a hypothesis formed after seeing the data as if you'd predicted it beforehand. Psychologist Norbert Kerr named the pattern in 1998; in experimentation, it means quietly rewriting "why we ran this" once the result disappoints.
It rarely looks like outright dishonesty. It looks like:
- Slicing the data into segments after the fact until one shows a positive result, then reporting that segment as the headline finding.
- Redefining the success metric after launch because the pre-registered one came back flat.
- Relabeling a clean null as "inconclusive, let's rerun it" specifically to avoid writing up a loss.
- Quietly extending a test's runtime past its planned stopping point because the trend line looked like it might turn positive.
None of these require a bad actor — they're the natural output of a team still scored on win count instead of learning quality. Four safeguards keep it out of a program:
- Pre-register the hypothesis, primary metric,
MDE, and stopping rule before the experiment ships, in writing, before anyone sees a result. - Separate confirmatory tests from exploratory segment digging, and label exploratory findings as hypothesis-generating — worth a follow-up test, not proof on their own.
- Cap the number of secondary metrics interpreted per test, or apply a statistical correction, so a "significant" result in metric #14 doesn't get promoted to the headline.
- Have someone other than the experiment owner sign off on the readout before any post-hoc segment cut gets added to the story.
Anchoring the hypothesis to something real before the test ships is the strongest defense of all. A hypothesis built around the Jobs to Be Done framework — "customers hire this flow to do X, so removing friction at step Y should increase completion" — is falsifiable on its own terms. It doesn't need a post-hoc segment to justify why it was worth running, which removes most of the temptation to go looking for one.
The Post-Experiment Readout Template That Makes Losses Useful
A post-experiment readout only earns its keep if it's filled out identically for wins, losses, and flat results — otherwise you get great documentation for the 20–30% that won and nothing for the rest. The template below takes about ten minutes and produces a record someone can actually find six months later.
Most teams already have a version of this template for wins; the gap is almost always that nobody bothers filling it out for a loss. That asymmetry, more than any lack of rigor, is why a program's actual learning rate is usually invisible even to the people running it.
- Hypothesis, verbatim — copied from the pre-registration, not rewritten after the fact.
- Primary metric,
MDE, sample size, and planned duration — the design that was committed to before launch. - Result — the actual numbers, the confidence interval, and an outcome classification: win, confirmed negative, true null, or inconclusive.
- What we now believe that we didn't before — one or two sentences, written even when the answer is "the mechanism we assumed doesn't apply here."
- What happens next — kill, iterate on the hypothesis, escalate to a bigger bet, or park with a stated reason.
- Who else should see this — which other team's roadmap or backlog this result should touch, even if it's not their experiment.
The habit that actually sustains this is treating a documented null the same as a shipped feature for velocity purposes. Reframing experiment velocity as shipping learnings, not just shipping winners is what makes readouts worth the ten minutes — a team that counts closed, well-documented questions toward its output has no incentive to bury the ones that didn't win.
This is also where a lightweight system matters, because the readout only works if the hypothesis is captured before the result, not reconstructed from memory afterward.
Prodinja's Journal is built around Assumption entries for exactly this: you record the assumption and the outcome that would falsify it up front, so when the test closes, the null result lands as a documented entry linked back to that assumption instead of a line nobody writes up. It doesn't grade the experiment for you; it just keeps the "before" honest enough that the "after" can't quietly change.
The Rituals That Turn Null Results Into Institutional Memory
Culture change happens through repeated, visible rituals, not a single all-hands slide about "failing forward." Growth teams that sustain a genuine null-results culture typically run three habits on a schedule: a recurring review of every closed test regardless of outcome, public win-rate reporting that includes the losses, and performance evaluation that rewards readout quality over raw win count.
A program's health shows up in the ratio of learning rate to win rate, not in the win rate alone — a team that closes clean answers on 90% of its tests is succeeding, even if only a quarter of them "win."
The experiment graveyard review
Once a month, walk through every experiment that closed since the last review — wins, confirmed negatives, and true nulls together, in the order they closed, not sorted to lead with the winners. Read the "what we now believe" line out loud for each one.
Map closed questions to the part of the customer journey they addressed, so the team can see which stages are well-understood and which are still mostly open, independent of whether any single test there won.
Public, honest win-rate reporting
Report win rate, learning rate, and experiment count together on the same dashboard leadership sees — never win rate alone. A win rate that looks great in isolation but comes with a low learning rate and a tiny sample of tests usually means a team is cherry-picking safe bets, not running a healthy program.
Evaluating people on the readout, not the scoreboard
This is where the growth PM vs. core PM skill stack genuinely diverges from a typical roadmap-execution role: a growth PM who runs twelve well-designed tests and confirms nine nulls has had a better quarter than one who ran three tests and got lucky twice. Calibrate performance reviews to hypothesis quality, statistical rigor, and readout completeness — not the number in the win column.
Naming the quarter's most useful loss
At the same cadence as celebrating the biggest win, name the null result or confirmed negative that changed the most future decisions — not the one with the funniest story, the one that closed the most expensive open question. Making this a recurring agenda item, not a one-off gesture, is what signals to the rest of the team that a rigorous loss counts as real output, on par with a lift.
Key Takeaways
- A 20–30% win rate is normal, not disappointing — benchmarks from Kohavi's Microsoft research and Booking.com's public experimentation talks both land well under 50%, and a win rate near 50% often signals timid bets or a loose
MDE, not skill. - Separate "failed" from "null." A confirmed negative or a true null is a closed, useful question; only an inconclusive, underpowered test is a genuine waste.
- Track
learning ratealongside win rate — the share of experiments that produced a decision-worthy answer — so the program's health isn't judged on wins alone. - HARKing is the biggest threat to an honest culture. Pre-register the hypothesis, metric, and stopping rule before launch, and keep exploratory segment digging clearly separate from confirmatory claims.
- Use one readout template for every outcome, filled out identically for wins and losses, so the record is searchable six months later.
- Rituals sustain the culture more than announcements do — a recurring graveyard review, transparent win-rate dashboards, and performance evaluation on readout quality all outlast any single "embrace failure" memo.
- Capture the hypothesis before the result, whether in a shared doc or a tool built for it, so the "before" can't quietly get rewritten once the "after" disappoints.
Frequently Asked Questions
What is a good win rate for a growth experimentation program?
Most mature programs land between 20% and 30%, based on publicly shared figures from Microsoft's experimentation research; Booking.com's testing program has described figures closer to 10%. A win rate consistently near 50% more often points to overly cautious tests or a loose MDE than genuine skill.
What's the difference between a failed experiment and a null result?
A failed experiment is one that didn't produce a usable answer — usually underpowered, contaminated, or stopped too early. A null result is a valid, adequately powered test that found no statistically significant effect, which is a real, useful answer, not a failure.
How do you stop a team from HARKing after a disappointing test?
Pre-register the hypothesis, primary metric, MDE, and stopping rule in writing before the test launches, and require sign-off from someone other than the experiment owner before any post-hoc segment cut gets added to the readout. Label exploratory findings as hypothesis-generating, never as confirmed proof.
Should you rerun an experiment that came back null?
Only if the original test was underpowered, contaminated, or stopped early — in which case it wasn't a valid null in the first place. A properly powered true null is a closed question; rerunning it without new evidence or a materially different hypothesis usually just spends traffic you could use on an untested idea.
How do you keep a team motivated when most experiments lose?
Report learning rate and experiment count alongside win rate so the scoreboard reflects the actual job, and evaluate individuals on hypothesis quality and readout completeness rather than raw wins. A recurring review that reads every closed test's "what we now believe" line out loud, regardless of outcome, does more for morale than any single pep talk.