A statistically significant lift is not proof you found a real effect—it is a claim that has to survive attack from peeking, underpowered samples, novelty effects, Simpson's paradox, and cherry-picked segments before it deserves belief. Treat a "winning" test as a hypothesis to interrogate, not a verdict to celebrate, until it survives a deliberate pre-analysis review.
Quick Answer: A winning test earns belief only after you've tried to kill it—check for peeking, underpowered samples, novelty decay, Simpson's paradox, and cherry-picked segments before you ship the result as truth.
The Shift: From Verdict to Hypothesis Under Attack
Most teams read a green metric on the experimentation dashboard and move straight to a rollout ticket. The more rigorous move is spending as much energy trying to disprove the result as you spent designing the test, because a p-value crossing 0.05 tells you an effect is unlikely under the null hypothesis—not that your story about why it happened is correct.
This is the same posture that makes adversarial thinking in product decisions useful everywhere else in the job: premortems, pricing changes, roadmap bets. Applied to experimentation, it means a positive result triggers a second process, not a celebration thread. Ron Kohavi, who ran experimentation platforms at Microsoft and Airbnb and co-authored Trustworthy Online Controlled Experiments, has spent years documenting how often "wins" don't replicate once teams look closer.
Kohavi's field is built around a principle known as Twyman's law: any statistic that looks interesting or surprising is probably wrong, or at least measuring something other than what you think.
Twyman's law, as applied in experimentation: the more a result looks like exactly what you hoped for, the more scrutiny it needs—not less.
A 40% lift on a small UI tweak should make you suspicious before it makes you happy. Real effects at that scale are rare; instrumentation bugs, bot traffic, and broken randomization are common. Treating the surprising win as the thing most likely to be an artifact—rather than the thing most likely to be your best idea yet—is the entire mental shift this article asks you to make.
Five Ways Your Experiment Lies to You
Every positive result can fail for one of five structural reasons, and each one produces a dashboard that looks identical to a genuine win. Knowing the five gives you a checklist instead of a vague feeling that something's off. None of them require bad faith—they happen inside well-run tests, run by careful teams, on legitimate data.
| Failure mode | What it looks like on the dashboard | What actually happened |
|---|---|---|
| Peeking | Significant result checked and stopped early | Repeated looks inflate the false-positive rate far above 5% |
| Underpowered sample | A large lift, wide confidence interval | Too few users to distinguish signal from noise |
| Novelty effect | Strong lift in week one, fading by week three | Users reacting to change itself, not to value |
| Simpson's paradox | Aggregate lift, but reverses within key segments | A confound (traffic mix, device, cohort size) flips the sign |
| Cherry-picked segment | One segment "wins" out of a dozen tested | Multiple comparisons guarantee a false positive somewhere |
Peeking: Stopping the Moment It Looks Good
Peeking is checking your dashboard daily and stopping the test the moment p < 0.05 appears, rather than at the pre-committed sample size or duration. Every additional look is another chance for random noise to cross the significance line, and standard significance tests assume you looked exactly once.
Researchers at Optimizely—Ramesh Johari, Leo Pekelis, and David Walsh, in their work on "always valid" sequential inference—showed that continuous monitoring against a fixed 0.05 threshold can push the real false-positive rate several times higher than the nominal rate, depending on how often and how long you peek. The fix isn't "never look"; it's using a sequential testing method built for repeated looks, or pre-registering a fixed sample size and duration and holding to it.
Underpowered Samples: Noise Wearing a Signal's Clothes
An underpowered test doesn't fail quietly—it often reports a large, exciting lift, because small samples produce wide, noisy estimates, and only the noisy draws that happen to clear significance get reported at all. Jacob Cohen's foundational work on statistical power (1988) is still the reference point: power depends on sample size, baseline rate, and the size of the effect you're trying to detect, and skimping on any one distorts the others.
The table below shows how sample-size requirements move with baseline conversion rate and the relative lift you're hoping to detect, using the standard two-proportion power formula at 80% power and a 0.05 significance threshold. Treat these as directional, not exact for your metric.
| Baseline rate | Relative lift to detect | Approx. sample size needed per variant |
|---|---|---|
| 5% | 10% | ~30,000 |
| 5% | 20% | ~7,600 |
| 5% | 30% | ~3,400 |
| 20% | 10% | ~6,400 |
| 20% | 20% | ~1,600 |
| 20% | 30% | ~710 |
Small relative lifts on low-baseline metrics need enormous samples. If your test ran at a fraction of the required size and still reported a big win, underpowering—not a genuine effect—is the more likely explanation.
Novelty Effects: New Beats Better, Briefly
A novelty effect is a lift driven by users noticing and exploring something new, not by the change actually serving their underlying job better. It decays over days or weeks as the new becomes routine, which is exactly why a test window that's too short mistakes curiosity for value.
This is easiest to catch by asking what job the change was hired to do. If a redesigned button gets more clicks in week one but the underlying job to be done hasn't changed, the extra clicks are exploration, not progress. Mapping how the reaction evolves across a customer journey emotion curve—excitement at first exposure settling into a steady-state emotional baseline—helps separate a real improvement from a shiny distraction.
Simpson's Paradox: When the Whole Lies About the Parts
Simpson's paradox is when an aggregate result reverses once you split the data by a real confounding variable, most often traffic mix, device type, or user tenure. The classic illustration is the 1973 UC Berkeley graduate admissions case, published by Bickel, Hammel, and O'Connell in Science: the university's overall admit rate looked meaningfully higher for men than women.
Yet when broken out department by department, women were admitted at the same rate or higher in most departments—they had simply applied more heavily to departments that were competitive for everyone.
The same mechanic shows up in product tests: a variant can win in aggregate purely because it happened to get more traffic from an easier-to-convert segment (mobile app, returning users, a specific geography), not because it's a better experience for anyone in particular.
Cherry-Picked Segments: The Win You Went Looking For
Cherry-picking is running a segment cut after the fact until one of them clears significance, then reporting that one. Test twenty segments at a 5% significance threshold and, on pure chance, roughly one will look like a winner even with zero real effect—that's what a 5% false-positive rate means at scale.
Before trusting a segment-level win, check whether the effect size stands up to a revenue hawk's critique of what it's actually worth. A segment "win" that only exists at the p<0.05 line, in a slice nobody predicted in advance, and that moves a metric nobody was optimizing for, is the profile of a false positive dressed as a discovery.
The Reversal: When a Winner Wears Off
A result can pass every significance check and still be wrong, because significance answers "is this probably not chance?" while durability answers "will this hold up next month?" Those are different questions, and only one of them shows up in a standard test-result email.
Consider a shape common enough in experimentation postmortems to have a name: a redesigned onboarding flow reports a strong activation lift in its first week or two of exposure. The team calls the test, ships it, and moves to the next priority. Weeks later, a routine metrics review shows activation has drifted back toward the old baseline—the "winner" quietly stopped winning once its glossy-new period ended.
This pattern is exactly what Kohavi's writing on primacy and novelty effects warns about: early-exposure reactions to any visible change—positive or negative—tend to be larger than the change's steady-state effect, and they fade at different rates depending on how often users encounter the changed surface. A control-group habit change (returning users treating the "old" version as newly unfamiliar) can distort the comparison in the opposite direction. Both failure modes point to the same fix.
- Re-run or extend the window past the point where daily-active users have cycled through the change at least twice.
- Segment by first-exposure date so you can compare early adopters' week-one reaction to their week-four reaction directly.
- Watch the control group too—a "novel" control experience for returning users can shrink the apparent lift for the wrong reason.
- Treat the first report as provisional in the memo itself, with a scheduled re-check date, not as a closed decision.
The Pre-Analysis Interrogation Checklist
Run this checklist before you write the results memo, not after someone questions it. Each item targets one of the five failure modes above, and most take minutes to check against data you already have.
- Was the sample size and duration set before the test started, and did you hold to it rather than stopping early on a good-looking number?
- Did the realized sample size meet the power calculation for the minimum effect size you cared about detecting?
- How many total looks at the data happened, and was the significance threshold adjusted for repeated checks (sequential correction) or held to a single pre-registered check?
- Does the lift hold when the test window is extended past the first exposure period for the average user?
- Does the result hold when broken out by device, traffic source, geography, and user tenure, or does it reverse in any major slice?
- Was this segment or metric decided on before the test ran, or found by scanning results after the fact?
- How many other segments or metrics were tested in the same experiment, and does the "win" survive a multiple-comparisons correction?
- Do the guardrail metrics (support tickets, refunds, latency, unsubscribes) stay flat, or does the primary metric win at a hidden cost?
- Would this result survive being explained to someone paid to disbelieve it, before it survives being explained to someone hoping it's true?
That last question is the whole discipline compressed into one sentence. A result that only survives friendly scrutiny hasn't been tested yet—it's been confirmed.
Red-Teaming the Memo Before You Ship the Belief
The fastest way to apply this checklist without relying on your own blind spots is to have something else argue against your result first. A pre-analysis checklist catches mechanical errors; a dedicated adversarial pass catches the interpretive ones—the story you told yourself about why the number moved.
None of this replaces statistical rigor upstream. It's a second, adversarial pass on top of a properly designed test—the layer that catches the interpretation errors a clean p-value can't catch on its own.
Key Takeaways
- A significant result answers one narrow question—is this probably not chance—not whether your explanation for it is correct or whether it will last.
- Peeking inflates false positives far past the nominal 5% rate; fix it with a pre-registered sample size or a sequential testing method built for repeated looks.
- Underpowered tests tend to report bigger, more exciting lifts, not smaller ones, because only the noisy draws that clear significance get reported.
- Novelty effects fade; re-check any early win after users have had time to stop noticing the change is new.
- Simpson's paradox can flip an aggregate result when a confound like traffic mix or device type differs between variants—always check the major segment cuts.
- Cherry-picked segments are a multiple-comparisons problem in disguise; a "win" found by scanning after the fact needs a much higher bar than one predicted in advance.
- The most useful question before shipping a belief is whether the result would survive being explained to someone paid to disbelieve it.
Frequently Asked Questions
How do I know if my A/B test result is a false positive?
Check the five failure modes directly: whether you peeked before the pre-set sample size, whether the realized sample met your power calculation, whether the lift decays over an extended window, whether it reverses in major segments, and whether it was a segment found before or after the fact. A result that fails any of these needs more evidence before you act on it.
What is a novelty effect in A/B testing?
A novelty effect is a temporary lift caused by users reacting to something being new or different, not to it genuinely serving them better—it typically fades over days or weeks as the change becomes routine. The fix is extending the test window and comparing early-exposure behavior to steady-state behavior for the same cohort.
What is Simpson's paradox and why does it matter for experiments?
Simpson's paradox is when a trend that holds in an aggregate dataset reverses or disappears once the data is split by a real confounding variable, most often because the groups being compared differ in composition (traffic mix, device, tenure). It matters because an experiment can show a clean aggregate win that is entirely explained by which segment got more traffic, not by the variant being better.
How much sample size do I need before trusting a test result?
It depends on your baseline rate and the size of the effect you're trying to detect—smaller relative lifts on lower-baseline metrics require dramatically larger samples, sometimes tens of thousands of users per variant. Run a power calculation before the test starts, and treat any "win" reported on a sample well below that number with active suspicion.
How long should I wait before trusting a winning test?
Long enough that users who first saw the change have had multiple opportunities to encounter it again, so an initial novelty reaction has time to settle into steady-state behavior—often a minimum of two to four full user cycles through the changed experience, not just the point where significance was first crossed. If the lift shrinks meaningfully over that extension, the original number was measuring curiosity, not durable value.