Experiment Analysis for PMs: Statistics You Actually Need

Statistically significant means the difference you saw is unlikely to be random noise — not that it's important, permanent, or ready to ship. For experiment analysis, you need four ideas: p-values, confidence intervals, statistical power, and minimum detectable effect. Everything else — t-tests, Bayesian priors, sequential testing — is implementation detail your experimentation platform already handles.

Quick Answer: Statistical significance (typically p < 0.05) tells you a result probably isn't random noise. It does not tell you the effect is large, durable, or worth shipping — that call needs the effect size, the confidence interval, and the assumption you logged before you ran the test.

What "Statistically Significant" Actually Means

A result is statistically significant when the gap between variants is unlikely to have happened by chance, usually judged against a 5% threshold (p < 0.05). That threshold says nothing about how big the effect is, whether it will hold up next month, or whether it's worth an engineering sprint — three separate questions PMs often conflate into one.

The 5% convention traces back to statistician Ronald Fisher, who proposed it in the 1920s as a rough, practical cutoff — not a law of nature carved anywhere. Nearly a century later, the American Statistical Association felt compelled to issue a 2016 statement warning that p-values were being systematically misread across science and industry, urging people to stop treating "p < 0.05" as a stand-in for "true" or "important."

A p-value answers one narrow question: if there were truly no difference between your variants, how surprising would data like this be? It is not the probability that your hypothesis is correct. It says nothing about effect size, and it can't tell you why a metric moved.

Here's what a low p-value does not mean, despite how often it gets read that way:

• It does not mean the effect is large — a tiny, commercially irrelevant lift can still be "significant" given enough traffic.
• It does not mean the result will replicate — a 5% false-positive rate means roughly 1 in 20 truly-null tests will look significant by chance alone.
• It does not mean your hypothesis was right for the reason you think — a real correlation inside one test still isn't causal proof of the mechanism behind it.

Good experiment analysis pairs the p-value with the effect size and treats both as inputs to a business decision, not the decision itself. That distinction is also where most data-driven decision-making practices quietly break down — teams stop at "is it significant" and skip "does it actually matter."

The Four Numbers Worth Knowing

Four stats concepts cover almost every PM decision in experiment analysis: p-value, confidence interval, statistical power, and minimum detectable effect. Learn what decision each one supports, and you can read most results dashboards without waiting on a data scientist to translate for you.

The confidence interval matters more than most PMs give it credit for. A test can report "+3% conversion, p = 0.04" — technically significant — with a 95% CI of [0.1%, 5.9%]. That's a real effect, but the interval, not the p-value, tells you the actual range of outcomes you're betting on.

Power and MDE are pre-test decisions, not post-test readings. The concept of power comes from Jerzy Neyman and Egon Pearson's 1930s hypothesis-testing framework, which added control over false negatives to Fisher's original method. Before you launch, size the test properly:

1. Decide the smallest lift that would actually be worth shipping — that's your MDE.
2. Estimate your baseline conversion rate and available weekly traffic.
3. Run those through a sample-size calculator (most experimentation platforms build one in) to get the required sample per variant.
4. Check that number against your real traffic — if the required run time is 11 weeks and your roadmap allows 2, you're not testing anything; you're guessing with extra steps.

Skipping this step is why so many tests get labeled "inconclusive" — they were never powered to detect the effect the team actually cared about. That's a vanity-metric trap in disguise: a test that reaches a verdict quickly, rather than the right verdict on the metric that matters.

The confidence level you pick is also a trade-off, not a default. 95% is the common convention, but it's arbitrary in the same way Fisher's 5% threshold is. A lower bar (90%) reaches a verdict faster with more false positives; a higher bar (99%) is slower but stricter — appropriate for a change that's expensive or risky to reverse, like a pricing test.

A Bayesian alternative exists too, and some experimentation platforms offer it alongside classic significance testing. Instead of asking "how surprising is this data if there's truly no effect," it asks "given this data, how likely is each possible effect size" — a more direct answer to what PMs actually want to know. The underlying math differs, but the same discipline applies either way: don't stop early, and don't mistake a probability statement for a guarantee.

The Traps That Wreck an Honest Read

Most bad experiment reads aren't caused by bad math — they're caused by process shortcuts that quietly invalidate math that was otherwise correct. Five traps are common enough to check on every single test: peeking, sample ratio mismatch, Simpson's paradox, metric shopping, and the novelty effect.

Peeking is checking your dashboard daily and stopping the moment it crosses p < 0.05. Ronny Kohavi ran experimentation platforms at Microsoft and Airbnb and co-authored Trustworthy Online Controlled Experiments. He has written extensively about how repeated early looks inflate the real false-positive rate well past the nominal 5% — in some published analyses, past 20-30% for teams that check often and stop at the first significant read. Fix it by pre-committing to a sample size or duration, or by using a platform with sequential-testing correction built in.

Sample ratio mismatch (SRM) is when your control-versus-variant traffic split doesn't match what you configured — 55/45 instead of 50/50, say. Kohavi's research treats SRM as a near-automatic reason to distrust every downstream metric in that test, because it usually signals a bug in randomization or logging rather than a real effect. Some teams run a periodic A/A test — both sides identical — purely to confirm their pipeline reports roughly a 5% false-positive rate at baseline before trusting it on anything real.

Simpson's paradox is when a trend reverses once you split the same data by segment. A variant can win overall while losing in every individual segment — mobile, desktop, new users, returning users — because the segments are weighted differently between control and variant.

A simplified, hypothetical illustration makes the mechanism concrete:

The variant loses in both segments individually, yet wins overall — purely because randomization happened to send it a traffic mix skewed toward the higher-converting returning-user segment. This is exactly where cohort analysis earns its keep: always check the top-line result against the same breakdown by cohort before trusting it.

Metric shopping — scanning a dozen secondary metrics and reporting the one that happened to move — is a version of what statistician Andrew Gelman calls the "garden of forking paths." With enough metrics in play, some will look significant purely by chance, even with zero p-hacking intent. Pick your primary metric before you launch, tie it to your north star metric, and treat everything else as directional context rather than proof.

The novelty effect distorts short tests specifically. A redesign can spike engagement simply because it's new — curious returning visitors click around, not because they prefer it — and that lift typically decays over the following weeks as everyone adjusts. Running a test for a single week risks mistaking curiosity for a durable preference shift, which is exactly why the duration check below matters as much as the sample-size check.

Kohavi's experimentation teams operated by a version of Twyman's Law: any statistic that looks surprisingly good is probably wrong. Treat a huge win as a reason to check your instrumentation before you celebrate it.

How to Read a Results Table Like a PM

Reading a results dashboard well means running the same short checklist every time, regardless of what the headline number claims. The goal is separating "this is real" from "this is real and worth acting on" — two different verdicts a single p-value can never give you on its own.

Before trusting any reported lift, walk through this order:

1. Check the sample ratio. Confirm the traffic split actually matches what you configured.
2. Check the duration. Did the test run at least one full business cycle — usually 1-2 weeks minimum, longer for anything with a weekly or monthly rhythm?
3. Check the confidence interval, not just the p-value. Is the plausible range narrow and meaningfully positive, or wide and centered near zero?
4. Check guardrail metrics. Did a "win" on the primary metric arrive with a hidden loss elsewhere — revenue up, support tickets up too?
5. Check the segment breakdown. Does the effect hold across your key cohorts, or does it disappear, or reverse, in one of them?

Once you've cleared that checklist, the decision usually falls into one of four buckets:

A "no effect" verdict is still information, not a wasted test. It tells you where in the customer journey this particular lever doesn't move behavior — which is often just as useful to know as where one does.

Where Analytics Mistakes Actually Start

Almost every bad experiment read traces back further than the dashboard, to an assumption that was never written down before the data arrived. Once you've seen the result, your brain quietly edits the hypothesis to fit it — a well-documented tendency psychologists call hindsight bias, and a core reason pre-registering a hypothesis is now standard practice in academic research design.

Without a logged hypothesis, three things reliably go wrong:

• HARKing (Hypothesizing After the Results are Known) — you tell a tidy story about why the metric moved that was never actually specified beforehand, only invented after the fact.
• Threshold drift — "significant" quietly becomes "directionally positive" the moment the original bar isn't met, because nobody wrote the original bar down anywhere.
• Silent scope creep — the metric you're now celebrating isn't the one the team originally set out to move, but nobody logged the switch when it happened.

The fix isn't more statistics training. It's writing the hypothesis, the metric, the expected direction, and the minimum effect worth acting on before you look at results — ideally the moment the idea is formed, not reconstructed afterward from memory or a Slack thread. A hypothesis rooted in an actual Jobs to Be Done insight, logged the moment you form it, is much harder to quietly reinterpret once the data disagrees with your gut.

Log the Assumption Before You Peek at the Data

Key Takeaways

• Significance answers one narrow question: is this probably not noise? It says nothing about size, durability, or business value — pair it with the confidence interval and effect size before deciding anything.
• Power and MDE are pre-test decisions. A test never sized to detect the effect you actually cared about will read "no result" no matter how good the underlying idea was.
• Peeking inflates false positives. Checking daily and stopping at the first significant read can push your real error rate several times past the nominal 5%.
• Check sample ratio mismatch and segment breakdowns before trusting a headline number — a real win at the top line can be a Simpson's-paradox illusion underneath.
• Pick one primary metric before you launch a test. Scanning a dozen secondary metrics afterward and reporting the one that moved is p-hacking with good intentions.
• Most bad reads start before the test even runs — with a hypothesis that was never actually written down, then quietly reshaped once the data arrived.