A flat overall result on an experiment often means two real, opposite effects canceled each other out — not that nothing happened. Heterogeneous treatment effects occur when a change helps one segment and hurts another; the average treatment effect blends both into a number that describes no actual user. The fix is asking "for whom did this work?" using segments defined before you saw the data, not ones you go fishing for afterward.
Quick Answer: When an A/B test average lands near zero, check pre-registered segments (not exploratory ones) for opposing effects. A change can lift new users +30% and drag power users -25% at the same time — the average shows +2% and everyone ships the "insignificant" test, or worse, cancels a real win alongside a real loss.
Why the Average Treatment Effect Can Actively Mislead
The average treatment effect (ATE) is a weighted mean across every user in the test, and a weighted mean has no obligation to resemble any individual's experience. If half your users gain and half lose by similar magnitude, the ATE reports something close to zero — a result teams routinely read as "no effect" instead of "two effects, canceling."
This isn't a statistical edge case; it's a structural property of averaging. Randomized controlled trials in medicine ran into the identical problem decades ago — a drug can help one genetic subgroup and do nothing (or harm) another, and the trial-wide average obscures both. The Cochrane Collaboration, which sets methodological standards for evidence synthesis in medicine, has long flagged this as why subgroup analysis exists as a discipline in the first place, alongside strict rules for when it's trustworthy.
Product experiments have the same shape, just faster and cheaper to run. A pricing page redesign might resonate with users who are three days into a trial and confuse users who are three months in. A notification frequency change might delight low-engagement users trying to re-engage and annoy high-engagement users who already have a working routine. None of this shows up in p < 0.05 on the topline metric.
The Simpson's Paradox Version of This Problem
Simpson's Paradox is the extreme case: a trend can literally reverse once you split the data, not just get diluted. A treatment can look like it helps overall while actually hurting every single segment inside it, if segment sizes and baseline rates interact in the wrong way. It's rarer in clean A/B tests than in observational data, but the underlying mechanism — aggregation hiding a structural difference between groups — is exactly the mechanism behind ordinary heterogeneous effects, just dialed to its most dramatic setting.
A milder, more common version happens constantly in growth experiments: a feature is shipped broadly, one large segment is mildly negative, one smaller segment is strongly positive, and the topline number is a weighted average that undersells the strong positive because that segment is a minority of traffic.
How to Read a Flat Result Before Assuming "No Effect"
A flat or barely-significant topline result deserves a segment check before it gets logged as a null result — but only against segments you named in your pre-registration, decided by a hypothesis about why effects might differ, not a dashboard fishing expedition. The distinction between the two is the entire discipline.
Start by revisiting your original hypothesis, the one you should have written before the test launched. If you followed a rigorous process for how to write a testable product hypothesis, you likely already stated a mechanism — "new users lack context, so simplifying onboarding should help them specifically." That mechanism is your segment candidate list; you don't need to invent one after the fact.
Signs the Average Is Hiding Something
Watch for these patterns in your results dashboard:
- A statistically insignificant topline result paired with a plausible causal mechanism that would only apply to part of your user base (new vs. tenured, mobile vs. desktop, high-intent vs. browsing).
- Wide confidence intervals relative to the effect size — a symptom of variance that segment-level noise, not just sample size, can produce.
- A metric that moved in early cohorts but flattened as the test accumulated more mature users, suggesting a time-since-signup interaction.
- Qualitative feedback (support tickets, session recordings, journal entries) describing opposite reactions — some users delighted, others confused — for the same feature.
If your team already ran the numbers through a standard statistical significance check, treat that as necessary but not sufficient. A non-significant ATE with any of the above signs is a prompt to look one level deeper, not a verdict to close the ticket.
Pre-Planned vs. Exploratory Segmentation: The Line That Keeps You Honest
Pre-planned segmentation means the segments and the specific metrics they'll be evaluated on are written down before the experiment launches; exploratory segmentation means you're slicing the results after seeing them. The first is legitimate hypothesis testing. The second is a search process that will always find something — and that something is usually noise wearing a pattern's clothes.
The math behind why exploratory segmentation is dangerous is the same math behind p-hacking generally: run enough comparisons and some will clear a significance threshold by chance alone. Statistician John Ioannidis, whose 2005 paper "Why Most Published Research Findings Are False" became a foundational citation across clinical and social science, showed that flexibility in analysis choices — including which subgroups to test — is one of the largest drivers of false-positive findings in published literature. Slicing your A/B test results by ten different user attributes after the fact is running ten implicit hypothesis tests, without correcting for it.
A Practical Comparison
| Dimension | Pre-planned segmentation | Exploratory segmentation |
|---|---|---|
| When segments are chosen | Before the experiment launches | After results are in |
| Basis for choosing segments | A stated causal mechanism | Whatever variable happens to correlate |
| Number of comparisons | Small, fixed, declared upfront | Often unbounded — "let's try slicing by X" |
| Statistical correction needed | Minimal, since segments are few and justified | Significant (e.g., Bonferroni or false discovery rate control) |
| Appropriate use | Confirming or refuting a specific mechanism | Generating new hypotheses for a future test |
| Risk if treated as confirmatory | Low | High — false positives get shipped as real wins |
The honest use of exploratory segmentation isn't "don't do it" — it's treat anything you find as a new hypothesis, not a conclusion. A pattern discovered by slicing after the fact earns a follow-up test designed specifically to confirm it, ideally on fresh traffic. Shipping a targeted rollout on the strength of a single exploratory slice is how teams end up chasing noise for a quarter.
Turning a Confirmed Segment Win Into a Targeted Rollout
Once a segment effect is confirmed — not just glimpsed in one test, but replicated or strongly pre-specified — a targeted rollout can capture value a global rollout would waste or actively destroy. If new users gain and power users lose, shipping to everyone nets out near zero and may quietly erode your best users' experience while you celebrate a flat topline as "safe."
This is where segment-aware experimentation earns its complexity budget. Consider the numbers behind that opening example:
| Segment | Share of users | Effect | Contribution to blended ATE |
|---|---|---|---|
| New users (< 14 days) | 30% | +30% | +9 points |
| Power users (> 6 months) | 25% | -25% | -6.25 points |
| Everyone else | 45% | ~0% | ~0 points |
| Blended average | 100% | ~+2.75% | Reads as marginal |
A team looking only at the blended row sees a weak, arguably-not-worth-shipping result. A team that segmented correctly sees a strong win available to nearly a third of its users, and a real cost to a segment worth protecting. The right decision isn't "ship" or "kill" — it's "ship to new users, hold or roll back for power users."
What Targeting Requires Operationally
- A reliable way to identify segment membership at the point of exposure — not just in retrospective analysis, since the rollout has to gate on the same attribute in real time.
- A monitoring plan for the excluded segment, so you'd notice if the untargeted group's baseline shifts and your segment definition goes stale.
- A re-test cadence — segment effects tied to product maturity or onboarding stage can decay as the product itself evolves, so a targeting rule isn't permanent.
- Clear documentation of why the segment was targeted, tied back to the original hypothesis, so a future team doesn't undo it without understanding the mechanism.
Teams that skip step 4 are the ones who, eighteen months later, "simplify" the codebase by removing a targeting rule nobody remembers the reason for — and quietly give back the win.
Defining Segments Honestly, Before You Ever Look at Results
The best segment hypotheses come from understanding why different users would react differently, not from a list of every attribute your analytics schema happens to track. A demographic or usage-based split invented on the spot is a much weaker foundation than a split grounded in what the user is actually trying to accomplish.
This is the same logic behind Jobs to Be Done thinking: users in fundamentally different circumstances — different jobs, different contexts, different points of struggle — are the users most likely to respond to the same change in opposite directions. Someone hiring your product to solve an urgent, first-time problem is doing a different job than someone using it as an established daily habit, and a UI change that helps one can easily hinder the other.
Mapping the emotional highs and lows across a customer journey is a related way to surface where a segment split is likely to matter — a moment that's confusing early in the journey and routine later in it is exactly the kind of point where an experiment's effect should be expected to diverge by tenure or familiarity, and naming that moment before you test is what makes the resulting segmentation defensible rather than convenient.
This is also where Prodinja's Customer Jobs tool fits into the workflow, if you're using it: it's designed to help you map users by the job and context they're in before you design the experiment, so the segments you pre-register come from a stated hypothesis about differing needs — not from a post-hoc scan of whatever columns exist in your event table. Principled segmentation starts with knowing your users' jobs, not with a GROUP BY clause.
Key Takeaways
- A flat average treatment effect can hide two real, opposite effects that cancel out mathematically without canceling out in reality for either group of users.
- Heterogeneous treatment effects are common in product experiments wherever users differ in tenure, intent, or context — not an edge case reserved for medical trials.
- Pre-planned segmentation, chosen before launch and grounded in a causal mechanism, is legitimate; slicing results after the fact to find a flattering subgroup is exploratory and prone to false positives.
- Simpson's Paradox is the extreme version of this problem — a topline trend can literally reverse once segments are examined, not just get diluted.
- A confirmed segment win justifies a targeted rollout, which can capture value a blanket global rollout would waste or actively cancel out against a losing segment.
- Exploratory findings are hypotheses for a future test, not conclusions — they need a dedicated confirmatory test before they justify a product decision.
- Grounding segments in user jobs and journey context, rather than convenient analytics attributes, is what keeps segmentation principled instead of a search for a flattering number.
Frequently Asked Questions
What is a heterogeneous treatment effect in A/B testing?
A heterogeneous treatment effect is when an experiment's impact differs meaningfully across subgroups of users instead of applying uniformly. One segment might see a strong positive effect, another a negative one, and the overall average blends both into a number that misrepresents everyone's actual experience.
How is segment analysis different from p-hacking?
Segment analysis is legitimate when the segments are decided before the experiment runs and tied to a specific causal hypothesis about why they'd differ. It becomes p-hacking when segments are chosen after seeing results, by testing many possible slices until one looks significant, without correcting for the number of comparisons made.
Can I trust a segment effect I found by exploring the data after the test ended?
Not on its own. Treat an exploratory finding as a new hypothesis worth testing, not a confirmed result — run a dedicated follow-up experiment, ideally on fresh traffic, before making a rollout decision based on it. Refer back to your original experimentation guide for how to structure that confirmatory test properly.
Why would a targeted rollout beat rolling a feature out to everyone?
If a change helps one segment and hurts another, a global rollout nets out the two effects and can leave you worse off with your most valuable users while claiming a "neutral" result. A targeted rollout ships the change only to the segment where it's confirmed to help, protecting the segment where it doesn't.
How many segments should I pre-register before running a test?
Keep the list short and mechanism-driven — typically two to four segments tied directly to your product hypothesis, not every attribute available in your analytics schema. Setting up the test correctly from the start, following a disciplined ab test setup process, makes it far easier to keep this list honest and small.