Statistical Significance Without a Stats Degree: A/B Testing for Growth PMs

An A/B test result is "statistically significant" when the gap between variants is unlikely to be random noise — the p-value, the probability of seeing that gap if there's truly no difference, falls under a threshold fixed before the test started, typically 5%. Skip that pre-registration step, and "significant" just means you found noise that happened to clear an arbitrary bar.

Quick answer: Significance is only meaningful if you fixed your sample size, primary metric, and p-value threshold before the test ran. Checking results daily and stopping the moment a number looks good — "peeking" — can inflate your true false-positive rate several times over, turning routine noise into a fake win.

What "Statistically Significant" Actually Means for a Growth Test

A p-value answers one narrow question: if there were truly no difference between control and variant, how often would you see a gap this large by chance alone? A low p-value (conventionally under 0.05) means that outcome would be rare under pure chance — not that your feature definitely works.

That distinction matters more than it sounds. Every test carries two possible failure modes, and growth PMs need both in their working vocabulary:

• Type I error (false positive): you declare a winner that isn't real. Convention caps this at 5% (alpha = 0.05) — you're accepting that 1 in 20 "wins" run on pure noise will still clear the bar.
• Type II error (false negative): you miss a real effect because the test wasn't sensitive enough to catch it. Power (conventionally 80%) is your defense here — the probability your test detects a true effect if one exists.

This alpha/power framework traces back to statisticians Jerzy Neyman and Egon Pearson, whose 1933 hypothesis-testing framework still underlies every A/B tool's "significant" badge. The 80% power convention itself comes from psychologist Jacob Cohen's 1988 work on statistical power — a widely adopted rule of thumb, not a law of nature.

Here's the shift most PMs never quite make: significance isn't something you discover in a dashboard. It's something you decide before you look — threshold, metric, and sample size are commitments made at launch, not judgments made at the end. That's worth internalizing early if you're building out the growth PM role as your specialty, since statistical literacy is one of the clearest lines in the growth PM vs core PM skill stack — a core PM can often lean on qualitative judgment; a growth PM's calls live or die by these numbers.

Power and Sample Size: The Rule of Thumb That Keeps Tests Honest

Your sample size determines whether a real effect is even detectable — run a test too small, and a genuine 10% lift will look like noise almost as often as it looks like a win. The rule of thumb: the smaller the effect you want to catch, the more visitors you need, and the relationship is roughly quadratic, not linear.

A common approximation for a two-variant test at 80% power and 5% significance is:

n per variant ≈ 16 × p × (1 − p) ÷ (minimum detectable effect)²

Where p is your baseline conversion rate and the minimum detectable effect (MDE) is the absolute lift you actually care about catching. Halve the MDE you want to detect and you roughly quadruple the required sample — the single most underestimated fact in growth experimentation.

Figures are per-variant, order-of-magnitude estimates from the standard sample-size formula above — a sanity check, not a spec. Run your exact baseline and MDE through a dedicated sample-size calculator (Evan Miller's widely used public one is a good default) before you launch.

Read the table sideways and the lesson is blunt: a low-traffic page with a low baseline rate chasing a small lift needs tens of thousands of visitors per variant — often more traffic than the page sees in a quarter. That's a real constraint on what's testable at all. It's part of why experiment velocity is a growth PM's real leverage: the skill is picking tests your traffic can actually power, not just tests that would be nice to run.

Before you launch, write down three numbers: your baseline rate, your minimum detectable effect (the smallest lift worth shipping for), and the sample size that combination requires. If your traffic can't reach that sample size in a reasonable window, the test isn't ready — shrink the scope, raise the MDE, or find a higher-traffic surface instead.

The Peeking Problem: Why Checking Results Daily Breaks the Math

Peeking — watching a live dashboard and stopping the test the moment p < 0.05 — is the single most common way growth teams manufacture false wins. Each look at interim data is itself a chance for a false positive, and those chances compound across every check made before the planned end date.

A 5% significance threshold assumes you check the result exactly once, at a sample size fixed in advance. Check five times over the life of the test and stop at the first "significant" reading, and your real false-positive rate is no longer 5% — you've effectively given noise five separate chances to clear the bar instead of one.

Directional figures reflecting repeated-significance-testing analysis, popularized in statistician Evan Miller's widely cited 2010 essay on the topic — not your exact numbers, but the shape holds: more peeks, more false positives, every time.

This is why experimentation researchers Ron Kohavi, Diane Tang, and Ya Xu, in their 2020 book Trustworthy Online Controlled Experiments, treat premature stopping as one of the most common integrity failures in real experimentation programs — not a beginner mistake, a structural one experienced teams still make under launch-date pressure.

None of this means you can't look at your dashboard. It means the number you look at mid-test isn't the number that decides anything — only the pre-committed sample size and metric get to make that call. If your team genuinely needs faster reads, adopt a sequential testing method built for continuous monitoring (several experimentation platforms now ship one), rather than eyeballing a fixed-horizon test more often than its design allows.

Pre-Registering the Metric and Threshold Before You Run the Test

Pre-registration means writing down your primary metric, minimum detectable effect, sample size, and significance threshold before the test goes live, and treating that document as binding. It's the practice that turns "we found something interesting" into "we found what we said we'd look for."

The habit isn't unique to product teams. Psychology's well-documented replication crisis — the 2015 Open Science Collaboration project, associated with researcher Brian Nosek, reproduced only a minority of a large sample of published findings when independent teams re-ran the original studies. The cause traced substantially to researchers testing many outcomes, then reporting whichever crossed a significance line after the fact. Pre-registration became the field's standard response.

Growth teams hit the same trap under a different name: metric fishing. Launch a test, watch ten dashboards, and something will eventually wiggle past p < 0.05 by pure chance — the more metrics you scan, the more likely one clears the bar for no real reason. This is the multiple comparisons problem, named for the correction (the Bonferroni method) statisticians use to compensate for it.

The fix is structural, not statistical:

1. Name one primary metric before launch — the single number that decides ship or kill.
2. Tie it to the job the feature is actually meant to do. A checkout redesign meant to reduce cart abandonment shouldn't get judged on session length just because session length happened to move. Grounding the metric in the job the customer is hiring the feature for keeps you from swapping in whichever number looks best after the fact.
3. Log secondary metrics as context, not competing win conditions. They can inform the next hypothesis; they don't get veto power over this one.
4. Assign ownership of that metric before the test, not after. If a test spans multiple funnel stages, decide upfront whose metric it is — this is exactly the discipline behind clean funnel ownership as a growth PM's real leverage: a shared metric with no named owner is how post-hoc metric-swapping happens in the first place.

Reading the Result Without Fooling Yourself: False Wins and Novelty Effects

A "significant" result at the end of a properly powered, pre-registered test is genuinely informative — but it still isn't the same as a durable, shippable win. Two more traps wait even after the sample-size and pre-registration work is done correctly.

The first is the novelty effect: a genuine short-term lift caused by users noticing something new, not something better. A redesigned checkout might spike clicks purely because returning users are curious, then decay back to baseline within weeks as the new becomes routine.

This is one reason where in the customer journey a test runs matters — a change touching a highly habitual, frequently repeated step is far more novelty-prone than one touching a rare, once-a-quarter action.

The second is calling a winner early because the trend "looks" like it's heading there. A test at 60% of its planned sample size showing a promising but not-yet-significant gap isn't a preview of the final answer — it's an incomplete read that can narrow, reverse, or vanish entirely before the test reaches its committed sample size. Regression to the mean is a real, well-documented statistical phenomenon, not just a cautionary phrase.

Practical checks before you trust a result enough to ship it:

• Did it hit the pre-committed sample size, or just a moment where the p-value happened to dip low?
• Does the effect matter in absolute business terms, not just statistically? A significant 0.2% lift on a low-value flow can be real and still not worth shipping — statistical significance and practical significance are different questions.
• Does it hold across a full weekly cycle (or more), so day-of-week and traffic-mix effects wash out rather than skew the read?
• Would it survive an A/A test — the same variant run against itself, to confirm your tooling isn't generating false positives before you trust the A/B read?

Make Pre-Registration a Habit, Not a Willpower Exercise

Knowing you should fix the metric and threshold before a test doesn't make it happen reliably under deadline pressure. The habit tends to erode exactly when a launch date is looming and "just a quick peek" feels harmless — the practice that survives contact with a real roadmap is one captured in writing before the test starts, not one that depends on remembering to resist temptation later.

This is the specific gap Prodinja's Journal is built to close: an Assumption entry captures the metric you're betting on and the threshold that decides success up front, as part of documenting the assumption itself, so the criteria exist in writing before any dashboard-watching begins. It doesn't run the statistics for you — it's designed to make fixing significance criteria before you peek the path of least resistance, instead of a step that gets skipped when a deadline is close.

Key Takeaways

• Significance is a pre-test decision, not a post-test discovery. Fix your metric, sample size, and threshold before launch — everything after that is just reading the number you already committed to.
• Sample size scales roughly with the square of the effect you want to detect. Halving your minimum detectable effect roughly quadruples the visitors you need per variant.
• Peeking inflates false positives, often severely. Checking daily and stopping at the first good-looking p-value can push your true error rate well past the 5% you think you're getting.
• One primary metric, chosen upfront, prevents metric fishing. Scanning many dashboards after the fact all but guarantees something will look "significant" by pure chance.
• Statistical significance and practical significance are separate questions. A real, detectable effect can still be too small to be worth the engineering cost of shipping it.
• Novelty effects and regression to the mean can both masquerade as real wins. Let a test run its full committed duration, ideally across full weekly cycles, before trusting the read.