Statistical Significance Without a Stats Degree: The Five Numbers Every PM Needs

Statistical significance for product managers means one thing: a claim that an observed difference probably isn't random noise, nothing more. It says nothing about whether the difference is big enough to matter to your business. Five numbers, read together, tell you whether to trust a result and whether to act on it: baseline rate, effect size, significance, power, and sample size.

Quick Answer: Significance answers "is this noise?" Effect size answers "is this big?" You need both, plus enough sample size and power to trust the first number, before you ship anything based on an experiment readout.

Most PMs learn statistics backward: formulas first, intuition never. That's why a p-value of 0.03 gets treated as a trophy instead of what it actually is, a statement about the probability of noise given a specific assumption. This piece skips the math and gives you the mental model, in plain language, using one worked example you'll recognize from your own dashboards.

Why "Statistically Significant" Isn't the Verdict You Think It Is

Statistical significance tells you whether an observed gap is likely to be real rather than random fluctuation. It does not tell you whether that gap is large enough to justify shipping, and it does not tell you whether you measured the right thing. Significance is a noise filter, not a business judgment.

Here's the shift that matters: every metric bounces around even when nothing changes. Run the same page twice with no differences and your conversion rate will still differ slightly between the two groups, purely from who happened to show up. Significance testing is a way of asking "is the gap I'm seeing bigger than the gap I'd expect from randomness alone?"

That's a narrow, technical question. It says nothing about:

• Whether the effect is large enough to be worth the engineering cost of shipping it
• Whether the metric you tested is the metric that actually matters to the business
• Whether the result will hold up next quarter, in a different market, or at a different scale
• Whether you ran the test correctly in the first place

Ron Kohavi, who ran experimentation at Microsoft and Airbnb and co-authored Trustworthy Online Controlled Experiments, has written extensively about how teams conflate "significant" with "important." His guidance, echoed across the experimentation literature, is blunt: treat the p-value as a gate you must pass before discussing business impact, not as the impact itself. Confusing the two is the single most common statistical error in product organizations, and it's why this article exists.

Number One: Baseline Rate, the Number Everything Else Is Relative To

The baseline rate is your current metric value before the experiment, the number every other statistic in this article is measured against. Get it wrong, misestimate it, or measure it over the wrong window, and every downstream calculation inherits the error.

Say your checkout conversion rate is 4%. That 4% isn't just context, it's load-bearing. It determines:

1. How much room there is to move. A metric near 50% has more room to swing than one near 1% or near 99%, which changes how sample size behaves.
2. What counts as a meaningful lift. A 0.5-point absolute lift on a 4% baseline is a 12.5% relative improvement. The same 0.5-point lift on a 40% baseline is barely a 1% relative change.
3. How volatile the metric naturally is. Rare events (a 1% baseline) produce noisier percentage estimates from the same sample size than common ones.

Teams get this wrong by pulling baseline from the wrong period, a holiday week, a launch spike, a stale dashboard, and then running an experiment against a number that was never accurate to begin with. Before you write a hypothesis, pull a trailing 4-8 week baseline and confirm it's stable. If you haven't formalized what a good hypothesis statement even looks like, how to write a testable product hypothesis is the place to start before the baseline conversation even happens.

Number Two: Effect Size, the Number That Actually Tells You If It Matters

Effect size is the magnitude of the difference you observed, whether measured as an absolute percentage-point change or a relative percentage lift, and it's the number that should drive your ship decision, not the p-value. Significance tells you the effect is probably real; effect size tells you whether it's worth anything.

This is the core distinction the whole article turns on: you can have a statistically significant result with an effect size too small to matter, and you can have a large, meaningful effect size that fails to reach significance because your sample was too small to detect it reliably. Both situations happen constantly. Neither is a contradiction, they're two different questions.

Only scenario C gives you both pieces you need. Scenario A is the trap most teams fall into, chasing a "win" that's statistically defensible but commercially irrelevant. Before you even run the test, decide what effect size would be worth shipping, sometimes called the minimum detectable effect or practical significance threshold. That number should come from a business conversation, not a statistics textbook: what lift, in dollars, retention, or activation, justifies the engineering and opportunity cost of this change?

Number Three: Statistical Significance, and What the P-Value Actually Says

Statistical significance, expressed through the p-value, tells you the probability of seeing a result at least this extreme if there were truly no difference between your groups. A p-value of 0.03 means: if the change genuinely did nothing, you'd still see a gap this large or larger about 3% of the time by chance alone.

That is a narrower, weirder claim than most people think it is. It is not:

• The probability that your hypothesis is true
• The probability that the effect is real
• A measure of how big or important the effect is
• Proof of anything, ever

Most teams set a significance threshold, commonly p < 0.05, meaning they'll accept a 5% chance of falsely declaring a winner when there was actually no difference (a false positive, or Type I error). That threshold is a convention, not a law of nature. Ronald Fisher, who popularized the 0.05 cutoff in agricultural research in the 1920s, described it as a rough, practical guideline for his era, not a universal truth. Teams running many experiments per year sometimes tighten it to reduce false positives; teams that value speed over precision sometimes loosen it deliberately.

The American Statistical Association issued a formal statement in 2016 warning against exactly this conflation, cautioning that a p-value alone does not measure the size of an effect or the importance of a result, and that decisions shouldn't rest on whether a p-value crosses a threshold.

One more nuance worth internalizing: the p-value is calculated under the assumption that nothing is different, then asks how surprising your data would be under that assumption. If the data would be very surprising under "nothing changed," you reject that assumption. That's the entire logical structure, and it's why significance is fundamentally a statement about noise, not about your product.

Number Four: Power, the Number That Tells You If You'd Even Notice a Real Effect

Statistical power is the probability that your test will detect a real effect of a given size, if one truly exists, given your current sample size and variability. Low power means you could run a test on a genuinely good change and walk away concluding "no difference," not because there wasn't one, but because your test wasn't sensitive enough to see it.

This is the number PMs skip most often, and it's the one that quietly invalidates the most experiments. A common convention targets 80% power, meaning if the true effect is exactly the size you're designing for, you'll correctly detect it 80% of the time. The other 20% of the time, a real effect slips through as a false negative (Type II error), and the team concludes the idea didn't work when it actually did.

Power depends on three things pulling against each other:

• Sample size: more traffic, more power, all else equal
• The effect size you're trying to detect: smaller effects need dramatically more traffic to detect reliably
• Metric variability: noisier metrics (long-tail revenue, session duration) need more traffic than clean binary metrics (did they convert, yes or no)

The practical failure mode: a team runs a test for two weeks, sees "no significant difference," and kills a genuinely good idea, when the real problem was that the test was underpowered to detect an effect that size in that timeframe. Before concluding a test failed, ask whether it had enough power to succeed in the first place. Andrew Gelman, a Columbia statistician who writes extensively on this exact failure mode, has documented how underpowered studies systematically produce both false negatives and inflated, unreliable effect-size estimates when they do reach significance, a double bind worth remembering.

Number Five: Sample Size, the Number That Ties the Other Four Together

Sample size is how many users or sessions you need in each group before you can trust the significance result, and it's calculated directly from the other four numbers: your baseline rate, the minimum effect size worth detecting, your significance threshold, and your target power. Change any one input and the required sample size moves with it.

The relationship is intuitive once you see the pattern:

This is why "just run it for two weeks" is not a sample size plan, it's a hope. The right sequence is: decide your baseline, decide the smallest effect size worth acting on, pick your significance and power targets, then calculate the sample size those choices require, before launch, not after. For the mechanics of that calculation, see how to calculate A/B test sample size; for getting the experiment structure right in the first place, setting up your first A/B test covers the setup decisions that determine whether your sample size math is even valid.

Decoding a Real Experiment Readout, Line by Line

A typical dashboard readout gives you numbers without context, and reading it correctly means walking through baseline, effect, significance, power, and sample size in order, not jumping straight to the p-value. Here's a readout close to what you'd actually see, decoded plainly.

The readout:

Walking through it:

1. Baseline (4.00%) — This is your reference point. A 0.35-point absolute move on this baseline is what "+8.75% relative" actually means.
2. Effect size (+0.35 pts, +8.75% relative) — Is this big enough to matter? That depends entirely on what you decided before the test, not on how the number feels now. If your team pre-committed to "a 5% relative lift or more justifies shipping," this clears the bar.
3. Significance (p = 0.04) — Under the assumption of no true difference, you'd see a gap this large about 4% of the time by chance. That clears a standard p < 0.05 threshold, but barely, this is not a blowout result.
4. Power — Not shown on the dashboard, which is exactly the problem. If this test was only powered to detect a 15% relative lift and it found 8.75%, you got lucky that it reached significance at all; a slightly smaller true effect might have been invisible to this test.
5. Sample size (18,000 per arm) — Was this the number the pre-test calculation called for, or the number that happened to accumulate by the time someone checked the dashboard? If the team peeked early and stopped at a "significant" moment, the 0.04 is likely optimistic, a well-documented issue called peeking or optional stopping.

The honest read: this is a real, probably-not-noise effect, borderline in strength, that clears a pre-agreed significance bar. Whether to ship depends on whether +8.75% relative was already defined as meaningful, and whether the team resisted the urge to stop the test the moment it looked good.

Where Prodinja Fits: Making Effect Size Concrete Before You Even Test

None of this is really about math, it's about connecting a metric delta to a business outcome you actually care about, which is harder to internalize from a formula than from seeing it move. That's the gap Prodinja's Business Scenario tool is designed to close in miniature, letting you adjust a metric like conversion or retention and watch how that shift ripples into a modeled outcome, so "effect size" stops being an abstract percentage and starts being a number tied to something concrete.

It won't run your significance tests or calculate your sample size, that's still your experimentation platform's job. What it's designed to do is build the instinct this article is arguing for: significance is a filter for noise, effect size is the question of whether the result is worth having, and the two only become useful once you can picture what the effect size actually means for the business. Prodinja is currently an interactive prototype, so treat the scenario modeling as a way to rehearse that judgment, not as a live statistical engine.

That same instinct, connecting a number to a decision, is also what separates a well-scoped experiment from a vague one in the first place. If your hypotheses tend to arrive without a clear "why this metric, why this magnitude," it's worth revisiting the complete guide to experimentation and grounding new tests in jobs to be done or a mapped customer journey so the effect size you're chasing connects to a real moment in the user's experience, not just a metric that moved.

Key Takeaways

• Significance is a noise filter, not a business verdict. A p-value tells you whether a gap is probably real; it says nothing about whether the gap is big enough to matter.
• Baseline rate is the foundation every other number depends on. Pull it from a stable, recent window, not a holiday spike or a stale dashboard.
• Effect size is the number that determines whether to ship. Decide your minimum meaningful effect before the test starts, not after you see the result.
• Power determines whether your test could even detect success. An underpowered test that finds "no difference" may simply have been unable to see a real effect, not proof the idea failed.
• Sample size is derived from the other four numbers, not an arbitrary duration like "two weeks." Calculate it before launch.
• A statistically significant result with a trivial effect size is a common trap. Always ask "significant, and how big?" together, never one without the other.
• Peeking at results early and stopping the moment they look good inflates false positives. Commit to a sample size and a stop date before you start watching the dashboard.