Your First A/B Test: A Setup Checklist That Prevents Garbage Results

A trustworthy first A/B test comes down to four setup decisions made before launch: the unit of randomization, an exposure event proving someone actually saw the variant, one pre-registered primary metric, and a stop rule stating exactly when you'll look and what you'll do. Skip one, and the data is noise wearing a p-value.

Quick answer: Fix the unit of randomization, log exposure separately from assignment, choose one primary metric, and set a stop rule — all before launch. These four setup decisions, not the analysis afterward, are what make a first A/B test on a real product trustworthy.

Why Most First A/B Tests Fail at Setup, Not Analysis

Most first A/B tests don't die from a statistics mistake — they die from a setup mistake nobody checked before launch. The randomization leaked between arms, the "exposure" event fired for people who never saw the change, or someone peeked early and called it. The fix happens before a single data point exists.

Say you're a PM at a B2B SaaS company piloting a new onboarding checklist meant to lift activation — the share of new accounts that complete a core action within seven days. That's a good first test: one clear behavior, one clear metric, a contained blast radius. It's also exactly the kind of test that goes sideways on setup, not on the readout.

A "good first test" is only as trustworthy as its setup. A clean hypothesis doesn't protect you from a contaminated randomization or a missing exposure event.

Ron Kohavi's research from Microsoft's experimentation platform — documented in Trustworthy Online Controlled Experiments (Kohavi, Tang, and Xu, Cambridge University Press) — found that roughly a third of tested ideas move the metric in the intended direction, a third show no meaningful difference, and a third make things worse. That two-thirds-uncertain outcome only means anything if the measurement itself is sound.

A broken pipeline and a true null result look identical on a dashboard. Only trustworthy setup lets you tell them apart.

Learning how to set up an A/B test that survives scrutiny in the readout meeting comes down to four decisions, made in this order:

• Unit of randomization — what you split into control and treatment, and whether that split can leak between the two groups.
• Exposure event — the log line proving a specific user actually encountered the variant, not just that they were eligible for it.
• Primary metric — the single number that decides the test, named before anyone sees data.
• Stop rule — the sample size or date, plus the significance threshold, fixed before launch.

This is one slice of a broader discipline — our complete guide to product experimentation covers where testing fits into a roadmap alongside qualitative research and staged rollouts. The rest of this piece walks through each of the four decisions using the onboarding-checklist test as a running example.

Pick the Right Unit of Randomization

The unit of randomization is whatever entity you split into control and treatment — usually the user, the account, or the session — and it should match the level at which your product actually behaves as a single unit. Pick the wrong one and you contaminate both arms before the test starts collecting anything real.

For the onboarding checklist, the unit is the account, not the user. The checklist is a shared workspace experience: if you randomize at the user level, one teammate sees the checklist and another on the same account doesn't, in the same week, on the same team.

They'll notice, ask each other about it, and now your "control" group has been partially exposed to the treatment through a Slack message. That's contamination, and it biases the result toward showing no difference — even if the checklist genuinely works.

Before you can name the right unit, you need a hypothesis specific enough to force the question. Our guide on how to write a testable product hypothesis covers the exact phrasing that surfaces this decision early, instead of leaving it implicit until an analyst asks "wait, randomized by what?" three weeks in.

Log the Exposure Event, Not Just the Assignment

Exposure logging is the record that a specific user actually encountered the variant, timestamped at the moment it happened — distinct from assignment, which only records that a flag evaluated to "treatment." Analyze by assignment when a chunk of assigned users never saw the change, and a real effect dilutes until it looks like nothing happened.

This is the single most common setup gap in a first test, because assignment is easy — it's just a flag check — and exposure requires deliberate instrumentation. For the checklist test, log the moment the checklist widget actually renders in front of a user, not the moment the account gets bucketed into treatment. At minimum, capture:

• account_id (or user_id, matching your randomization unit)
• the variant assigned
• a timestamp of first exposure — not assignment
• the surface where exposure occurred (dashboard, in-app modal, email)
• a stable experiment_key and version, so a mid-test fix to the variant doesn't silently merge into the same bucket

Whether that log line fires client-side or server-side is itself an instrumentation decision with real consequences. Client-side events undercount anyone on a slow connection, an ad blocker, or a browser that never finishes rendering — which quietly skews your sample toward faster connections and newer devices.

Most trustworthy experimentation setups log assignment server-side and treat the client-side event as a secondary confirmation, not the source of truth. Our analytics instrumentation guide covers the event schema and pipeline discipline this depends on, and our technical foundations guide covers the client-versus-server tradeoffs in more depth.

If your only exposure signal is "the flag was on for this account," you don't have an exposure event — you have an assignment log wearing an exposure event's name.

Choose One Primary Metric and Ignore the Rest Until It Reads

A primary metric is the single, pre-declared number that decides whether the test wins, chosen before launch and left alone once data starts arriving. Track guardrail metrics alongside it, but only the primary metric gets to declare victory. Checking five metrics after the fact and reporting the best-looking one as "the result" is how noise gets published as a win.

The American Statistical Association's 2016 statement on statistical significance warned specifically against this pattern: testing many outcomes and highlighting only the significant one distorts the true false-positive rate of the entire exercise, even when each individual comparison looks clean.

A dashboard with fifteen metrics and no designated primary is an invitation to find one that moved by chance and call it the result.

For the checklist test, the primary metric is the 7-day activation rate — the percentage of new accounts completing the core action within seven days of signup, defined with an exact numerator, denominator, and time window before launch. Everything else supports that read without being allowed to override it.

Guardrails matter because a metric can win narrowly while quietly breaking something else — a checklist that lifts activation by nudging users through steps they don't understand, for instance, can also spike support tickets. That's not a false positive; it's a real effect with a cost you didn't price in until the guardrail flagged it.

Set the Stop Rule Before You Launch

A stop rule is the pre-committed answer to two questions, written down before launch: how much data before you look, and what happens for each possible result. Without one, stopping the moment a partial result "looks significant" — commonly called peeking — inflates your false-positive rate well past the 5% you think you're protecting.

Evan Miller's widely cited essay How Not to Run an A/B Test laid this out plainly over a decade ago: a test checked repeatedly and stopped at the first significant-looking peek behaves nothing like a test checked once at a pre-set endpoint, even though both use the identical significance formula. The math assumes one look. Peeking gives you many free rolls of the dice, and eventually one comes up "significant" by chance alone.

Two disciplined approaches exist:

1. Fixed-horizon testing — you declare a sample size or calendar duration upfront and don't touch the result until it's reached. This is the right default for a first test.
2. Sequential testing — statistical methods designed to let you check safely along the way, spending a pre-allocated error budget as you go. Worth adopting once your team runs enough concurrent tests to justify the tooling.

Teams that run experimentation at real scale build this discipline into the platform, not into individual willpower. Booking.com's experimentation team has publicly described running thousands of concurrent controlled tests with results locked until a pre-set sample size or duration is hit, specifically so no single person can eyeball a dashboard and call the game early.

A written stop rule specifies:

1. Minimum sample size per arm, calculated from your baseline rate and the MDE (minimum detectable effect) — a free sample-size calculator gets you there in minutes.
2. Minimum duration — at least one full weekly cycle, so a Tuesday-only read doesn't get generalized to "how the product performs."
3. The significance or confidence threshold you'll use to call it, decided before you have a stake in the answer.
4. What happens if it's inconclusive — extend once, with a new pre-set limit, or ship the safer variant based on guardrail metrics rather than re-running the same test hoping for a cleaner number.

For the checklist test: minimum 400 activated accounts per arm based on a baseline 22% activation rate and a minimum detectable effect of 4 points, minimum two full weeks to span a billing cycle, 95% confidence, and — if inconclusive — extend once for one more week before defaulting to "ship the safer variant."

The Pre-Launch A/B Test Checklist (and the Mistakes That Sink a Test)

A pre-launch checklist forces the four setup decisions into writing before a single user is exposed — unit of randomization, exposure logging, primary metric, and stop rule — plus the basic hygiene items first-time testers skip under launch pressure. None of these items are hard individually; the failure mode is skipping one because "we'll figure it out from the data."

The Checklist

1. Hypothesis written down in an if/then/because format, naming the unit it applies to.
2. Unit of randomization confirmed, checked against contamination risks (shared accounts, shared devices, shared inboxes).
3. Exposure event instrumented and tested in staging — confirm it fires for both variants, on the slowest realistic connection.
4. Primary metric named, with its exact definition (numerator, denominator, time window) visible to the whole team.
5. Guardrail metrics listed, with the threshold that would veto a "win."
6. Sample size and duration calculated from baseline rate and minimum detectable effect, before launch.
7. Stop rule documented: date or sample size, significance threshold, and the decision for each outcome.
8. Both variants QA'd against your design system, so a rendering bug in the treatment doesn't masquerade as a real effect — see our design systems guide for what "consistent" should mean across variants.
9. The variant reviewed for usability, not just novelty — different isn't automatically better; our product design and UX guide covers evaluating a design change before it becomes a test.
10. An owner named who can confirm at any point mid-test that it's still running as designed.
11. A rollback plan in case the treatment breaks something the guardrails didn't anticipate.

Common Mistakes That Turn a Test Into Noise

Changing the variant mid-flight deserves its own warning, because it feels harmless in the moment — "we just tightened the copy, it's basically the same experience." It isn't. Everyone bucketed into treatment before the edit saw version A; everyone after saw version B. Your "treatment" arm is now a blend of two different experiences, and no analysis afterward can cleanly separate them.

Documenting the Setup So Engineering Ships the Instrumentation, Not Just the Feature

Most of this checklist fails quietly because it lives in a Slack thread, a slide, or someone's memory — not in the spec engineering actually builds from. Prodinja's Spec Studio lets you document the test's primary metric, variants, and stop rule inside the same living PRD that defines the feature itself, so the build ticket includes the exposure event and guardrail thresholds alongside the UI change, not as an afterthought.

Because Spec Studio tracks changes as PR-style diffs, a variant edited mid-flight shows up as a visible change to the spec, rather than a silent edit nobody flags until the readout looks strange.

Key Takeaways

• Setup, not analysis, is where first A/B tests usually break. Fix the unit of randomization, the exposure event, the primary metric, and the stop rule before you write any experiment code.
• Match your randomization unit to how the product actually behaves. Shared, multi-seat products need account-level splits; single-user products can usually randomize at the user level.
• Assignment is not exposure. Log the moment someone actually encountered the variant, not just the moment a flag evaluated to "treatment."
• One primary metric decides the test. Guardrails can veto a win; diagnostics explain why; neither gets to declare victory on its own.
• Write the stop rule down before you have a stake in the answer. Peeking at partial results and stopping early is the single most common way a null result becomes a false "win."
• Changing the variant mid-flight silently corrupts the result — treat any tweak, however small, as a reason to restart the clock.
• Document the setup where engineering actually builds from, not in a side conversation that nobody can audit once the test is live.