You Can't Hotfix a Binary: Running A/B Tests When Your Code Is Frozen in the Store

A frozen mobile binary can't be patched mid-experiment the way a web page can, so every mobile A/B test must be built around server-side feature flags that toggle behavior already shipped inside the app. You decide variants at runtime from a config service, not at deploy time. Get the flag plumbing wrong and you're stuck running a test you can't stop, on a version mix you can't control.

Quick answer: Mobile experimentation can't rely on redeploys. Ship all variant logic behind server-controlled flags inside one binary, track app version as a segmentation variable (not noise), and never launch a client-only test without a server-side kill switch.

Why Mobile A/B Testing Is Structurally Different From Web

The core answer: a web team can rewrite a page and push it in minutes, but a mobile team ships a compiled binary that Apple and Google review, and users update on their own unpredictable schedule. This single fact reshapes how every mobile experiment must be designed, not just how it's deployed.

On web, an experimentation platform swaps HTML, CSS, or JS at request time. The "variant" and the "control" live on the same server, and a bad result gets fixed with a rollback deploy in seconds. Mobile has no equivalent instant undo. Once a binary is in someone's pocket, it does exactly what it was compiled to do until that person updates it — which, per multiple app analytics vendors including Adjust and installations data cited by RevenueCat, can take anywhere from days to many months depending on the platform and user segment.

That gap forces three structural constraints onto mobile experimentation that web teams rarely think about:

1. Review latency. Any new experiment variant that requires new code must survive app store review timelines before a single user sees it — often 24-48 hours on iOS, sometimes longer, plus your own QA cycle.
2. Version fragmentation. At any moment your install base spans several app versions, each with different bugs, different UI, and sometimes different event schemas.
3. No universal kill switch inside the code itself. Whatever ships is permanent for that binary unless you built an external lever to override it.

The Practical Fix: Move the Decision Server-Side

The direct answer: you solve all three constraints by never hardcoding the experiment outcome in the binary — instead, ship the capability for multiple behaviors and let a server call decide which one activates for a given user, session, or cohort.

This is why server-driven feature flagging (via tools like Firebase Remote Config, LaunchDarkly, Split, or a homegrown config endpoint) has become the default architecture for mobile growth teams, not an optional nicety.

Server-Side Feature Flags: The Only Reliable Lever You Have

The direct answer: server-side flags let you assign, target, ramp, and kill an experiment without touching the binary, because the app checks in with a config service at launch (and periodically) and renders whichever pre-built code path the server tells it to. Everything else — targeting rules, percentage rollouts, emergency shutoff — lives outside the app.

A workable pattern looks like this:

• Client ships both paths. The binary contains the control experience and the variant experience, compiled and QA'd together, before the experiment ever starts.
• A remote config call assigns the variant. On launch (and on a defined refresh interval), the client fetches an assignment: variant_a, variant_b, or control, keyed by a stable user or device ID so assignment is sticky.
• The app renders based on the flag, never a local default. No if (Math.random() > 0.5) logic baked into the binary — that's a client-only test, and it's the trap covered below.
• A remote kill switch exists independent of the experiment flag. If the variant crashes or tanks conversion, you flip one server value and every instance of the app reverts to control on its next check-in.

Build vs. Buy for the Flagging Layer

The third row is listed only to make a point: shipping a "test" as two separate App Store submissions isn't A/B testing, it's sequential launches with no control group. Treat it as a design smell, not a shortcut.

Version Fragmentation Is a Confound, Not Background Noise

The direct answer: because users update at different times, your experiment population is never a single app version — it's a moving blend of versions with different bugs, UI, and instrumentation, and if you don't segment by version you will misattribute version-driven effects to your treatment.

Say you launch a checkout-flow experiment inside v4.2. Within a week your population looks roughly like this:

If your analysis pools all of these together, a real treatment effect from v4.2 gets diluted by users who were never in the experiment at all, and a genuine version-4.0 bug (say, a broken analytics event) can masquerade as a treatment effect. Mixpanel, Amplitude, and most mobile analytics vendors recommend explicit version-level segmentation for exactly this reason.

How to Analyze Across Versions Without Fooling Yourself

• Filter your denominator to eligible versions only. Only include sessions from binaries that actually contain the flag check — older versions structurally cannot be "control" in any meaningful sense, they're just absent from the experiment.
• Report metrics per version, then pooled. A per-version breakdown surfaces version-specific bugs before they contaminate your headline number.
• Watch for update-adoption bias. Power users and early adopters update faster and behave differently than laggards — so early experiment data over-represents a non-representative slice of your base.
• Tag every event with the app version at time of firing, not the current version at analysis time, so historical exports stay honest even as new versions ship.

Treat app version the way a clinical trial treats site location: a variable you stratify on, not a nuisance you average away.

Minimum Sample Size Realities on Mobile

The direct answer: mobile experiments usually need larger minimum samples and longer run times than equivalent web tests, because per-user daily activity is lower, effect sizes on core metrics tend to be smaller, and version fragmentation effectively shrinks your true eligible population below your total install base.

Standard frequentist sample-size math (Evan Miller's widely used online calculators formalize this well) depends on baseline conversion rate, minimum detectable effect (MDE), and desired statistical power — typically 80% power at a 95% confidence level. Two mobile-specific realities push the required sample up:

1. Lower session frequency means it takes longer to accumulate the same number of exposures a web experiment would gather in days.
2. Effective population shrinkage. If only 55% of your install base is on an eligible version, your real sample size is little more than half your total daily actives, even though your dashboard shows the full number.

A Rough Framework for Setting Expectations

Nielsen Norman Group and other usability-research bodies have long cautioned against ending tests the moment they hit "statistical significance" mid-run — peeking inflates false-positive rates. Pre-register your minimum sample and stopping rule before the experiment goes live, and hold to it.

The Danger of Client-Only Tests You Can't Turn Off

The direct answer: a client-only test — one where the variant logic and the coin-flip both live in the shipped binary with no server override — is dangerous specifically because you lose the one thing that makes mobile experimentation survivable: the ability to stop it. Once that binary is in the wild, the test runs until users update, which could be months.

This is the scenario every mobile growth PM eventually inherits from a predecessor: a hardcoded if user.hash % 2 == 0 buried in the app, no remote flag, no analytics tagging which arm someone was in, discovered only after a support ticket spike. There's no rollback. You either wait out the natural update curve or ship an emergency release and go back through app store review again.

Before approving any mobile experiment, run this checklist:

• Is the variant decision made by a server call, not local device logic?
• Is there a kill switch flag independent of the experiment assignment flag?
• Does every exposure event log the flag value and app version together?
• Has QA verified the kill switch actually reverts behavior on a cold app start, not just a warm resume?
• Is there a documented owner who can flip the kill switch outside business hours?

Skipping any of these turns an experiment into a liability with a multi-month tail.

Deciding Which Bets Deserve a Server-Gated Experiment

The direct answer: server-gated experiments carry real engineering cost — flag plumbing, QA of both code paths, version-aware analysis — so not every idea deserves one, and a lightweight prioritization pass before committing engineering time saves you from running expensive tests on low-value bets.

This is where structured prioritization earns its keep before an experimentation backlog fills up with ideas nobody scored. Prodinja's RICE and Kano prioritization tools are designed to help you weigh a proposed mobile change's reach, impact, and confidence against the heavier setup cost of a proper server-gated test — versus cases where the honest answer is to just ship the change and watch aggregate metrics, because the expected value doesn't justify the experimentation overhead. Not every UI tweak needs a controlled test; some just need a good jobs-to-be-done rationale and a careful post-launch watch.

That triage decision matters more on mobile than web precisely because the cost of testing (flag infrastructure, dual-path QA, version-segmented analysis) is so much higher than the cost of testing on a redeployable web stack.

Key Takeaways

• Mobile binaries can't be hotfixed, so every experiment variant must ship inside one binary and be activated by a server-side flag, never hardcoded logic.
• Version fragmentation is a confound you must segment on, not noise you average away — filter analysis to eligible versions and tag every event with the firing version.
• Minimum sample sizes run larger on mobile than equivalent web tests because of lower session frequency and shrunken effective eligible populations.
• Client-only tests with no server override are the single riskiest pattern in mobile experimentation — always build a kill switch independent of the assignment flag.
• Not every idea deserves a server-gated experiment — score reach, impact, and confidence first, and reserve the heavier testing cost for bets that clear that bar.