Is It Really Personalized? Teardown Techniques for Telling ML From If-Statements

Most "AI-powered personalization" you encounter is a handful of if/then rules in a marketing wrapper — segment tags, purchase-history lookups, or basic collaborative filtering, not a model learning your individual behavior. You can tell the difference with a behavioral probe: create fresh accounts, deliberately diverge your actions, and time how fast and how specifically the output actually changes.

Quick answer: Assume any "personalized" feature is running rules until it proves otherwise. Create 2-3 clean test accounts, feed them identical then diverging behavior, and measure adaptation speed and specificity. Fast, generic shifts are cohort rules. Slow, "people like you" shifts are collaborative filtering. Only fine-grained, individually-traceable shifts that update within a session are a true per-user model.

What "Personalized" Actually Means — And Why Most Products Fake It

"Personalization" spans a wide spectrum, and most products marketed as "AI-personalized" sit at the cheap end of it: static segmentation or collaborative filtering, not a model trained on your individual signal. The word gets used identically for a SELECT * WHERE segment = 'high_value' rule and for a neural network updating weights on your clickstream in real time. Both ship the same landing-page copy: "personalized just for you."

There are four honest layers worth distinguishing before you evaluate any claim:

1. Static rules — content is the same for everyone, or hardcoded by a single attribute (new vs. returning, geography, device).
2. Cohort/segment rules — you're bucketed into one of a fixed number of groups (by plan tier, declared interest, an RFM score) and every member of that bucket sees identical treatment.
3. Collaborative filtering — output is derived from patterns across many users' behavior ("people who did X also did Y"), without a persistent model of you specifically.
4. True per-user models — a model trained or fine-tuned on your individual interaction history, updating as you generate new signal.

Only the last layer is what most people mean when they say "AI personalization." The other three are legitimate engineering choices — but they are not that, and conflating them is exactly the gap this teardown protocol is built to expose.

Assume Rules Until Proven Otherwise

The correct starting prior for any personalization claim is skepticism: assume it's rules, cohorts, or collaborative filtering, and make the product prove it's doing more. This isn't cynicism for its own sake — it's the cheaper, more common, more maintainable engineering choice, and regulators have started saying so out loud.

The U.S. Federal Trade Commission's business-guidance team has repeatedly warned companies against what it calls "AI washing" — advertising ordinary automated or rule-based logic as artificial intelligence. In a 2023 post titled "Keep your AI claims in check," FTC attorney Michael Atleson wrote that firms should be prepared to back up AI marketing claims with evidence.

Analysts covering enterprise software, including Gartner, have flagged the same underlying pattern for years: vendors relabel rule engines and lookup tables as "AI" because the term sells better than "if-statement."

There's a product logic behind the skepticism too, rooted in Jobs to Be Done. Users don't hire a product for its personalization architecture — they hire it to get a job done faster, and a well-tuned rule set can satisfy that job almost as well as a real model, at a fraction of the cost.

Our complete guide to Jobs to Be Done covers why the underlying mechanism rarely matters to the user — which is precisely why teams under-invest in the real thing and over-invest in the label.

Treat "personalized," "smart," and "AI-powered" as marketing claims requiring evidence — the same way you'd treat "clinically proven" on a supplement label.

The Teardown Protocol: Testing What Actually Adapts to You

You can distinguish real adaptation from a rules engine with a structured behavioral probe: build 2-3 clean identities, give them a shared baseline, then deliberately diverge their behavior and log what changes, how fast, and how specifically. This mirrors the rigor described in our teardown methodology that teaches, which treats every teardown as a testable hypothesis, not a vibe check.

Before you start, decide which competitor's claim is worth this effort — our guide to choosing what products to teardown walks through picking targets where the answer will actually change a decision.

Step 1: Build Clean, Isolated Test Accounts

Create at least two, ideally three, fresh accounts using different devices, IPs, and email domains. Cross-contamination (shared cookies, shared device fingerprints, logged-in-elsewhere sessions) is the single most common reason a teardown produces false positives for "personalization" that's actually just device-level tracking.

• Use separate browser profiles or a VPN to avoid shared IP-based cohorting.
• Avoid connecting any social/OAuth login that could merge identity graphs.
• Document account creation timestamps precisely — timing evidence matters later.

Step 2: Establish an Identical Baseline

Perform the exact same five to ten actions on every test account: same searches, same clicks, same dwell time as best you can control it. Screenshot or export the resulting recommendations, feed, or "for you" surface immediately after.

If the accounts already diverge at this stage, before you've done anything different, you've found either a cold-start default that varies randomly (a tell for A/B testing infrastructure, not personalization) or account metadata bleeding through (signup source, referral link, declared preferences at onboarding).

Step 3: Deliberately Diverge Behavior

Now split the accounts hard. One account behaves like persona A (say, price-sensitive, browses only the cheapest tier), one like persona B (browses premium, ignores discounts entirely), and a control account does nothing further. Vary behavior at different points in the customer journey — first session, day three, week two — since personalization claims often apply to only one moment in that journey rather than the whole arc; our complete guide to the customer journey is useful for mapping where those moments actually sit.

Step 4: Time the Adaptation

Log the exact interaction count and elapsed time before you see any change in output. This is the single highest-signal measurement in the whole protocol:

Step 5: Test Specificity, Not Just Presence

A change happening isn't proof of personalization — a specificity test is. Ask: did the output shift toward "people like you" (a cohort or CF signal) or toward your exact, idiosyncratic behavior (a per-user signal)? Do something contradictory — browse premium for a week, then suddenly buy only the cheapest item — and see whether the system updates on that single contradicting event or clings to the old pattern.

• Cohort/rule systems usually don't recover from a contradiction quickly; they wait for the next batch re-scoring.
• Collaborative filtering shifts you toward a different cohort, not toward your specific act.
• True per-user models tend to weight the most recent, most specific signal more heavily, sometimes overcorrecting visibly.

Reading the Signatures: Cohort Rules vs. Collaborative Filtering vs. True Per-User Models

Each personalization mechanism leaves a distinct evidence trail — adaptation speed, output specificity, cold-start behavior, and the language used to explain recommendations all differ predictably by type. Once you've run the protocol above, compare your notes against these signatures rather than guessing from a single data point.

The "customers who bought X also bought Y" phrasing is a particularly reliable tell — it's close to verbatim the framing Amazon engineers used in their landmark description of item-to-item collaborative filtering, published by Greg Linden, Brent Smith, and Jeremy York in IEEE Internet Computing in 2003. If a product's copy echoes that pattern, you're very likely looking at neighborhood-based filtering, not an individual model — a completely legitimate technique, just not the "AI trained on you" story being sold.

Collaborative filtering itself traces back further, to the GroupLens Research group at the University of Minnesota, whose early-1990s Usenet recommender (Resnick et al.) is generally credited as the origin of the modern "people like you" approach.

The Netflix Prize competition (2006–2009) popularized matrix factorization variants of the same idea, and Netflix's own engineering blog has since described blending that with contextual bandits for row-ordering — a hybrid worth knowing exists, because "hybrid" is often the honest answer to "is it real personalization," not a clean yes or no.

Common Tells That Give Away a Rules Engine

A handful of surface behaviors reliably expose a rules-or-cohort system even without the full multi-account protocol: identical recommendations across logged-out sessions on the same device, output that snaps back to a default after a short window, and language that names a segment rather than an action. Use these as fast triage before committing to the full teardown.

• The logged-out test. Clear cookies, reload. If "your personalized picks" look identical to what you saw signed in, it's IP- or device-cohort logic, not an account-bound model.
• The plateau test. Keep interacting for a long stretch. Real per-user systems keep shifting; rule systems hit a ceiling because there are only so many buckets to be sorted into.
• The vocabulary test. "Recommended for [demographic/tier]" language signals segmentation. "Because you watched/bought/searched [exact item]" signals item-level filtering at minimum.
• The reversal test. Undo your recent behavior (unfollow, remove from cart, mark "not interested"). Rule systems often keep showing the same bucket-driven content; true per-user systems visibly react.
• The new-account speed test. A brand-new account instantly getting eerily specific recommendations, with zero behavioral history, usually means it inherited signal from something other than your behavior — device fingerprinting, a shared IP, or a purchased/matched identity graph.

None of these alone is conclusive. Two or three together, logged consistently, build a case a marketing claim can't talk its way out of.

Recording the Evidence So Your Read Stays Honest

A teardown is only as trustworthy as its record — write down every account, every diverging action, and every timestamp as you go, not from memory afterward, or you'll unconsciously fill gaps with what the marketing copy told you to expect. This is the step most PMs skip, and it's the one that turns a credible-sounding hunch into a defensible competitive read.

Keep a running log with, at minimum: account ID, action taken, timestamp, and the resulting output (screenshot or text export). Our teardown note-capture system covers a structure that holds up across a multi-week probe, which matters here because collaborative-filtering and true-model signatures often only separate cleanly after several days of divergent behavior, not one sitting.

For a broader refresher on the discipline this whole approach sits inside, our complete guide to teardowns and case-study methodology is the right starting point before your next one.

Key Takeaways

• Default to skepticism. Assume "personalized" means rules, cohorts, or collaborative filtering until the evidence says otherwise — regulators like the FTC have explicitly flagged "AI washing" as a real, common pattern.
• Speed and specificity are your two core signals. Fast-but-generic changes point to cohort rules; slow-but-neighborhood-shaped changes point to collaborative filtering; fast-and-individually-traceable changes point to a true per-user model.
• Build clean test accounts before you test anything else. Cross-contaminated cookies, shared IPs, and merged identity graphs are the most common source of false-positive "personalization."
• The contradiction test is the highest-leverage single move. How a system reacts when you behave inconsistently reveals more than a week of consistent behavior does.
• Explanation copy is a legitimate evidence source. "Customers who bought X also bought Y" is a documented collaborative-filtering signature, not a guess.
• Most real systems are hybrids. Don't force a binary verdict — the honest finding is often "cohort rules for cold-start, collaborative filtering after that, no evidence of a true per-user model."
• Log as you go, not after. A timestamped record beats a memory-based summary every time you need to defend the read to a skeptical stakeholder.