Novelty & Primacy Effects in A/B Testing

A first-week lift can be a novelty effect (existing users reacting to change itself, not the change's value) or a primacy effect (new users need time to learn a flow, so early data underrepresents their true response). Both distort early reads. The fix isn't more significance testing — it's tracking time-to-stabilize by plotting lift over time and segmenting new versus tenured users before you call the result.

Quick Answer: A lift that shrinks steadily over 2-3 weeks is decaying novelty, not a real effect. Plot daily lift by user tenure; if the curve flattens toward zero, wait for the settled number before you ship.

Growth PMs live and die by the "ship it" decision, and nothing manufactures false confidence faster than a beautiful week-one dashboard. You hit p < 0.05 on day 4, the lift reads +18%, and the Slack thread is already drafting the launch announcement. Three weeks later the same metric is flat, or worse, negative. Nobody broke anything — the experiment just never finished revealing what it actually was.

This is not a statistics failure. Statistical significance tells you the difference you observed is unlikely to be random noise at the moment you measured it. It says nothing about whether that moment was representative of steady-state behavior. That's a separate validity question, and it's the one growth teams skip most often because dashboards are built to answer "is this significant," not "is this stable."

What Are Novelty and Primacy Effects in A/B Testing?

Novelty and primacy effects are validity threats that inflate or deflate an early experiment read because users are reacting to the change itself rather than its long-term value. Novelty effect: existing users notice something new and engage with it out of curiosity, temporarily. Primacy effect: users (often new ones) need time to learn a changed flow, so their early behavior undersells the eventual outcome.

Both effects distort the same underlying signal — how real users will behave once the change is just "how the product works" — but they pull in opposite directions, which is why they get confused with each other constantly.

Novelty effect: the curiosity spike that fades

Existing, tenured users notice a redesigned button, a new onboarding screen, or a repositioned CTA and click it simply because it's different. This is well documented in HCI research — Jakob Nielsen's usability writing has long flagged that any visible UI change generates a temporary attention bump independent of whether the change is actually better. The spike is real engagement, but it's engagement with novelty, not with the improvement you shipped.

Existing users over-engage with the new element in week one purely from curiosity.
The bump concentrates in users who were already active and already familiar with the old version.
It decays as the "new" thing becomes the normal thing — usually within one to three weeks depending on session frequency.

Primacy effect: the learning curve masquerading as a flat result

Primacy effect runs the opposite direction. A genuinely better checkout flow, for instance, can look neutral or worse in week one because users — especially new users — haven't yet learned the new pattern. Early friction from unfamiliarity suppresses a true positive effect until habituation catches up.

New users need repeat exposure before a changed flow's real ergonomics show up in behavior.
Early "no effect" or slightly negative reads can flip positive once the learning curve completes.
This is why killing a variant on a flat day-3 read is as dangerous as shipping one on an inflated day-3 win.

Both effects are forms of the same problem: early data is a biased sample of eventual behavior, biased in different directions depending on whether users are reacting to change-as-stimulus (novelty) or change-as-obstacle (primacy).

Why Time-to-Stabilize Matters as Much as Significance

A significant result measured too early can still be the wrong number, because significance describes confidence in a snapshot, not confidence that the snapshot represents steady state. The shift growth PMs need to make is treating "has this effect stabilized" as its own gate, sitting alongside — not subordinate to — the p-value.

Think of it as two independent questions an experiment has to answer before you act on it:

Question	What it tells you	What it misses
Is the lift statistically significant?	The observed gap is unlikely due to sampling noise at this point in time	Whether that point in time reflects long-run behavior
Has the lift stabilized over time?	Whether the effect size is holding steady across successive time windows	Nothing on its own — pairs with significance, doesn't replace it

A result can be significant and unstable (early novelty spike, still statistically real in week one) or stable and not yet significant (a small genuine effect that needs more days of data to clear the noise floor). You need both boxes checked, not one. Our guide to the five numbers that matter for statistical significance covers the confidence-interval side of this; time-to-stabilize is the complementary temporal side that a single-point significance check can't see.

How long is a typical novelty window?

There's no universal constant — it scales with how often users encounter the changed surface. A daily-use core flow (search, feed ranking) tends to stabilize faster, often within one to two weeks, because users accumulate exposures quickly. A low-frequency flow (annual billing settings, a once-a-quarter export feature) can take much longer simply because users haven't returned enough times to move past "notice the change" into "have an opinion about it." Ron Kohavi's writing on controlled experiments at Microsoft and Airbnb (in Trustworthy Online Controlled Experiments) is a widely cited source for treating novelty/primacy decay as a named, expected pattern rather than an anomaly — his guidance is to run experiments long enough to observe at least one full stabilization cycle for the surface in question, not a fixed universal day count.

How to Detect a Decaying Effect: Plot Lift Over Time

The single most reliable detection method is plotting daily (or weekly) lift as its own time series, rather than only checking the cumulative lift-to-date number that most dashboards default to. Cumulative metrics smear early spikes across the whole window, hiding decay behind an average that still looks positive.

Compute lift per time bucket (daily is usually granular enough) instead of only cumulative lift since experiment start.
Plot the buckets as a line chart, treatment lift over time, with the control baseline as a flat reference line at zero.
Look at the shape, not just the level. A flat line near a constant value is a stable effect. A line that starts high and slopes down toward zero (or toward a lower plateau) is decaying novelty.
Identify the plateau, if there is one. If the curve flattens at, say, +6% after starting at +20%, the settled effect is closer to +6% — that's the number to make your ship decision on, not the week-one +20%.
Watch for the inverse shape too — a line that starts near zero or negative and climbs is primacy resolving, and killing the test before the climb finishes throws away a real win.

A useful rule of thumb: don't trust a lift number until you've observed at least two consecutive stable-length windows (e.g., two full weeks) reporting similar values. One flat week could still be a transitional plateau; two in a row is a much stronger signal you've hit steady state.

Why segmenting new vs. tenured users clarifies the picture

Splitting the same time-series chart by user tenure separates the two effects that a blended chart conflates, because novelty and primacy live in different user populations. Novelty effect concentrates in existing/tenured users who have a "before" to compare against. Primacy effect concentrates in new users who are learning the flow for the first time with no prior baseline to be startled out of.

Tenured users' lift curve decaying toward zero → classic novelty; the change was noticed, not necessarily preferred.
New users' lift curve flat or negative early, then rising → primacy; give it more exposures before judging.
Both segments converging on a similar, stable positive number → the strongest evidence of a durable effect, because it's not an artifact of either population's transition state.

Building the Discipline Into Your Experiment Process

Preventing novelty/primacy misreads is a process fix, not a statistics fix — it means writing the stabilization check into how you design and gate experiments, before the first result ever appears on a dashboard. Retrofitting skepticism onto a dashboard that's already showing a tempting green number is much harder than deciding the rule in advance.

Write the stabilization criterion into the hypothesis before launch. A well-formed hypothesis states not just the expected direction but the conditions under which you'll trust the read — see our guide to writing a testable product hypothesis for the format.
Set the minimum runtime at experiment setup, not after seeing early results — deciding "we'll look at day 4" after day 4 already looks good is exactly backward. Our walkthrough on setting up your first A/B test covers pre-registering runtime alongside sample size.
Default new experiments to at least one full stabilization cycle for the surface being tested, using low-frequency surfaces' longer cycles as your baseline when unsure.
Build the time-series and segmentation views into your standard readout, not as a follow-up investigation you only run when a result looks suspiciously good.
Treat "the win faded" as a normal outcome to plan for, not a failure of the original experiment — it's exactly what the methodology is supposed to catch.

If any of this sounds like it overlaps with hypothesis quality generally, it does — this is one specific validity threat inside the broader discipline our complete guide to experimentation walks through end to end, alongside sample-size planning, novelty's cousin threats (seasonality, network effects), and analysis pitfalls.

Using a reminder to force the second look

Key Takeaways

Novelty effect inflates early results because existing users react to the change itself, not its lasting value — and it decays as the change becomes routine.
Primacy effect deflates early results because users, especially new ones, need time to learn a changed flow before their true response shows up.
Statistical significance and time-to-stabilize are separate checks — a result can be significant and still unstable, or stable and not yet significant.
Plot lift per time bucket, not just cumulative lift, and look for a decaying or rising shape rather than trusting a single point-in-time number.
Segment by user tenure to separate novelty (tenured users) from primacy (new users) in the same chart.
Set your minimum runtime and stabilization criterion before launch, not after an early result already looks appealing.
A reminder to re-check the experiment after the novelty window turns "we should watch this fade" into an action that actually happens.

Frequently Asked Questions

How long does a novelty effect typically last in A/B testing?

There's no fixed universal duration — it scales with how frequently users encounter the changed surface, typically one to two weeks for daily-use flows and considerably longer for low-frequency features. The reliable signal isn't a day count; it's watching the lift-over-time curve flatten across at least two consecutive stable windows.

How can I tell if my early A/B test win is a novelty effect or a real improvement?

Plot daily lift as a time series instead of relying on cumulative lift-to-date, and segment by user tenure. A curve that starts high and decays toward a lower plateau, concentrated in tenured users, points to novelty; a curve holding steady across two-plus consecutive windows in both new and tenured segments points to a real effect.

Does the novelty effect always fade, or can it convert into a genuine lasting improvement?

It can go either way — some initial curiosity converts into durable habit if the underlying change genuinely serves users better, while pure novelty decays fully back toward baseline. The only way to know which happened in your case is to keep measuring past the initial spike until the curve stabilizes, rather than assuming either outcome.

Should I extend an experiment if I suspect a primacy effect is suppressing the results?

Yes, if the flat or negative early read is concentrated in new users who haven't had repeat exposure yet — that's the classic signature of primacy resolving over time. Extend the runtime and re-check the new-user segment's curve specifically before deciding the change underperformed.

Is a statistically significant result ever wrong because of timing?

The statistic itself isn't wrong — it correctly describes the data collected — but decisions built on it can be wrong if that data was captured during a novelty spike or primacy dip rather than steady state. Treat significance and stabilization as two separate gates an experiment must clear before you act on it.