Ship More Experiments: Turning Opinions Into Testable Assumptions

Most test backlogs are opinion backlogs wearing lab coats. The fix is a falsifiable hypothesis with a stated mechanism, a minimum detectable effect, and a pre-committed decision rule, plus a portfolio view of every bet you're carrying. Do this and "run more tests" becomes "run the right tests, correctly."

Quick answer: Convert every test idea into a hypothesis with a change, an expected effect, a mechanism, and a threshold — then log it as an assumption, size it with a minimum detectable effect, and refuse to peek before the pre-registered sample size is hit.

Why Opinion-Shaped Ideas Kill Experimentation Velocity

An opinion-shaped idea ("let's try a sticky add-to-cart bar") isn't testable because it has no falsifiable claim attached. It can't lose. Without a stated mechanism and threshold, any result — flat, up, or down — gets rationalized into a story after the fact.

Real experimentation velocity isn't about running more tests per quarter. It's about running tests that produce a decision regardless of outcome. Ron Kohavi's work at Microsoft's Experimentation Platform team found that roughly a third of tested ideas show no measurable effect and another chunk actively hurt the metric they were meant to help — which only matters if you defined "hurt" and "help" in advance.

Three symptoms tell you your backlog is opinion-shaped rather than hypothesis-shaped:

• No mechanism. The idea states what to change but never why it should move behavior.
• No threshold. "See if it works" isn't a stopping rule; it's an invitation to peek until something looks good.
• No falsifiability. If every outcome (up, down, flat) can be explained as a win, the test wasn't a test.

The Cost of Untestable Backlogs

Untestable ideas don't just waste dev cycles — they corrode trust in the whole program. When stakeholders can't tell why a test shipped or what it disproved, experimentation starts to look like theater. That's the fastest way to lose executive air cover for a testing calendar.

The Hypothesis Template: Change, Effect, Mechanism, Threshold

A testable hypothesis names the exact change, predicts a directional effect on a named metric, states the behavioral mechanism, and commits to a numeric threshold before launch. Skip any one field and you've written an opinion, not a hypothesis.

Use this four-field template for every backlog item:

Notice the mechanism field is doing the real work. It forces you to articulate a theory of behavior, which is what makes a null result informative. If the mechanism was "reduces search cost" and the test is flat, you've learned something real about how people scan a mobile PDP — not just that "the button thing didn't work."

This discipline pairs naturally with a Jobs to be Done lens: a mechanism that isn't tied to the job the customer is trying to get done (find the product, judge fit, commit to purchase) is usually a decorative guess. Our Jobs to be Done complete guide walks through anchoring hypotheses in the forces that push and pull customers toward a purchase decision, which is a sturdier mechanism source than "competitors do this."

Minimum Detectable Effect: Size the Bet Before You Place It

The minimum detectable effect (MDE) is the smallest lift you actually need the test to be capable of finding, and it determines your required sample size before a single visitor is exposed. Skip this step and you'll either run underpowered tests that can't detect real effects, or run oversized tests that waste traffic proving something obvious.

MDE calculation depends on four inputs, and Growth PMs should be able to reason about all four without opening a stats textbook:

1. Baseline conversion rate — your current metric value (e.g., 3.2% checkout conversion).
2. Desired statistical power — typically 80%, the odds you'll detect a true effect if one exists.
3. Significance level (alpha) — typically 5%, your tolerance for false positives.
4. Traffic available — how many visitors/sessions you can realistically allocate per week.

The relationship that matters intuitively: smaller effects require exponentially more traffic to detect, not linearly more. Halving the MDE roughly quadruples the required sample size. This is why "let's just test everything" breaks down fast on any page below your top five traffic destinations.

A Practical MDE Gut-Check

Before writing code, ask: given our traffic, what's the smallest lift we could actually detect in four weeks? If the answer is bigger than any plausible effect from the proposed change, don't build it — the test can't tell you anything. This single question kills a surprising share of low-traffic backlog items before engineering time is spent.

Evan Miller's and Georgi Georgiev's published sample-size guidance both converge on the same operational point: calculate sample size from the MDE you care about, not from "how long marketing wants the test to run." Calendar-driven test durations are how underpowered tests enter production decisions disguised as evidence.

Sample-Size Discipline and the Discipline of Not Peeking

Sample-size discipline means committing to a pre-calculated visitor count (or duration) before launch and refusing to call a winner until you hit it — because checking results daily and stopping at the first significant-looking p-value inflates your false-positive rate far above the stated 5%.

This is the single most common way "successful" tests turn out to be noise. Peeking at results daily and stopping the moment p < 0.05 appears is a well-documented statistical trap: with continuous monitoring, the actual chance of a false positive can climb toward 20–30% instead of the intended 5%, depending on how often you check.

Three guardrails prevent this:

• Pre-register the stopping rule. Write the required sample size or duration into the test ticket before launch, and treat it as immutable unless the test is causing real harm.
• Use sequential testing methods if you must monitor continuously. Approaches like always-valid p-values or Bayesian sequential testing (surfaced in tools built on Kohavi's and Georgiev's public research) are designed for legitimate early stopping — ad hoc peeking is not.
• Separate "monitoring for harm" from "monitoring for a win." It's fine to watch for a metric craterin g so you can kill a broken test early. It's not fine to stop early because a positive result appeared.

The test that "won early" and the test that ran its full pre-registered duration are not the same evidence, even if both show p < 0.05.

Turning the Roadmap Into a Portfolio of Bets

Reframing the backlog as a portfolio of assumptions — each with a stated confidence, cost to test, and potential upside — turns "what should we test next" from a debate about taste into a prioritization exercise with visible tradeoffs. That's the same lens RICE scoring or Kano classification apply to features, just pointed at hypotheses instead.

An assumption log gives every test backlog item four columns beyond the hypothesis fields:

Ranking by expected information value (not just expected lift) surfaces a pattern fast: cheap, reversible, high-uncertainty tests near high-traffic pages usually beat expensive, low-uncertainty tests on low-traffic pages, even when the latter has a flashier projected number attached.

This portfolio thinking connects naturally to adjacent CRO surfaces. A hypothesis about search relevance benefits from the query-intent framing in our site search relevance guide; a hypothesis about recommendation placement should borrow the mechanism thinking in AI recommendations beyond "people also bought"; and anything touching the purchase moment should be weighed against the stakes discussion in checkout flow optimization, since checkout tests carry asymmetric downside if the mechanism is wrong.

Where Assumptions Get Lost

The most common failure mode isn't a bad hypothesis — it's a good hypothesis that nobody wrote down, so it either never gets tested or gets re-litigated from scratch six months later when someone has the same idea. An assumption log is only useful if it's actually consulted before backlog grooming, not archived in a slide deck.

Making the Assumption Log Something Your Team Actually Uses

Logging assumptions works only if capturing one is as fast as having the thought — otherwise the log decays into a once-a-quarter chore that misses most real ideas. This is the operational gap most CRO programs never close.

Key Takeaways

• Opinion-shaped ideas aren't testable — they lack a mechanism, a threshold, or a way to be wrong. Reject them from the backlog until they have all three.
• Use the four-field hypothesis template — change, expected effect, mechanism, threshold — for every backlog item, no exceptions.
• Calculate minimum detectable effect before writing code. If your traffic can't detect a plausible effect size, the test isn't ready to build.
• Pre-register sample size and refuse to peek for wins. Continuous monitoring without a sequential-testing method inflates false positives well beyond your stated significance level.
• Treat the roadmap as a portfolio of bets, ranked by expected information value (confidence × reach × reversibility), not just projected lift.
• Log assumptions the moment they're spoken, not after they're formalized — a backlog only reflects real bets if capture is frictionless.