Fake Doors, Wizards, and Concierges: Testing Assumptions Cheaply

A fake door test measures demand without building anything; a Wizard of Oz test measures value by faking the automation behind a real interface; a concierge MVP measures value and feasibility by delivering the outcome manually. Pick the test based on which assumption — demand, feasibility, or value — is most likely to kill the idea, and set your pass/fail line before you launch it.

Quick answer: Fake door tests measure demand (will anyone click?). Wizard of Oz and concierge tests measure value (will they actually use and pay for the outcome?). Landing-page tests measure demand plus early positioning. Run the cheapest test that can kill your riskiest assumption — and write the kill threshold down before you look at results.

Most teams skip this menu entirely and go straight to build, because "just ship a small version" feels faster than designing a test. It rarely is. A badly-scoped MVP still takes weeks, still confounds multiple assumptions at once, and still leaves you unable to say why it failed. The tests below are built to isolate one assumption at a time, cheaply enough to run in days.

What assumption are you actually testing?

Before choosing a test, name the assumption it needs to kill: demand ("will people want this?"), feasibility ("can we actually build/deliver this?"), or value ("will it be good enough that they keep using or paying for it?"). Most product failures trace back to skipping this step and testing the wrong risk with the wrong tool.

This is the core discipline behind lean product discovery: every experiment should map to a specific belief that, if wrong, sinks the idea. Teresa Torres's opportunity solution tree framing makes this explicit — an assumption test only earns its place on the tree if it's tied to a specific opportunity and a specific solution bet, not run in the abstract.

The three risks, defined

Eric Ries's Lean Startup popularized this triad, and it still holds up as the fastest sorting mechanism available:

Notice that demand tests and value tests aren't interchangeable. A fake door can get thousands of clicks and still fail on value the moment real humans use the real thing — clicking "I'm interested" costs nothing, so it only tells you people are curious, not that the solution works.

The fake door test: cheapest signal, weakest evidence

A fake door test puts a button, ad, or menu item for a feature that doesn't exist yet, then measures how many people click it. It retires the demand assumption cheaply — usually in a day of design work and a week of traffic — but it cannot tell you anything about value or feasibility, because nobody ever gets the thing they clicked for.

How to run one

1. Add the door — a "New: Bulk Export" button, a pricing tier that doesn't exist, a nav item leading to a feature.
2. Route clicks to a landing page or waitlist, not a 404 — capture intent and let people self-select into "notify me."
3. Set the threshold before launch — for example, "8% of users who see the button must click it, and at least 20% of clickers must submit an email," not "let's see what happens."
4. Run for a fixed window (typically 1-2 weeks of representative traffic) and stop — don't keep extending the test hoping for a better number.

What it retires: demand only. What it can't retire: whether the feature, once built, is good enough to keep people around. Treat a passing fake door test as "worth investigating further," never as "worth building."

A fake door test that can't fail isn't a test. If your click-through threshold is set so low that almost any traffic clears it, or if you never precommit a number, you've built a vanity metric, not an experiment.

Wizard of Oz test: real front end, human-powered back end

A Wizard of Oz test shows users a real, polished interface while a human secretly does the work the interface implies is automated. It's named for the film's man behind the curtain, and it's the single best way to test whether an automated experience delivers real value before you've built the automation. Zappos founder Nick Swinmurn ran an early version of this pattern in 1999 — photographing shoes from local stores and manually fulfilling orders placed on a website with zero inventory behind it, to test whether people would actually buy shoes online.

When to reach for it

Use a Wizard of Oz test specifically when the risk is "can this be automated well enough to feel good," not "does anyone want it." It's the right tool when:

• The proposed feature is an AI or algorithmic layer (recommendations, matching, summarization, scoring)
• Manually producing the output is possible, even if slow or unscalable
• You need to learn what "good" output looks like before specifying it for engineering

Worked example. Say your assumption is: "PMs will act on an AI-generated weekly risk digest for their stakeholder relationships." Instead of building the model, a PM manually reviews five pilot users' stakeholder notes each Friday and sends a formatted digest email that looks machine-generated. Threshold set in advance: "At least 3 of 5 users reply or take a visible action within 48 hours, for 3 consecutive weeks." Two of five acting inconsistently, or one enthusiastic user carrying the average, is a fail — the threshold exists precisely so a shrug doesn't get rounded up to a win.

Concierge MVP: no illusion, just manual delivery

A concierge MVP delivers the full outcome by hand, with users knowing a human is doing the work — no illusion of automation, unlike Wizard of Oz. It retires value and feasibility together: you learn whether the outcome is wanted enough to pay for, and you learn exactly what the manual process requires, which becomes your build spec if it works.

Airbnb's founders famously did this in reverse-engineered form — personally photographing hosts' apartments and staying in touch with early guests, learning what made a listing convert before any of it was self-serve. The distinction from Wizard of Oz matters: concierge is transparent about the human, which makes it better suited to relationship-heavy or trust-sensitive offers, where hiding the human would itself be a confound.

Concierge vs. Wizard of Oz, side by side

Set the threshold and the disclosure plan together. If a Wizard of Oz test succeeds, tell users the automation was manual before scaling to real customers — undisclosed illusion tests that continue past the pilot stage cross from research into deception.

Landing-page tests and the falsifiability trap

A landing-page test publishes a page describing the product or feature, drives targeted traffic to it, and measures signups, waitlist joins, or purchase-intent clicks. It's a demand test like the fake door, but richer — you can test messaging variants, pricing anchors, and positioning simultaneously, not just "does a button get clicked."

The falsifiability trap

The single biggest failure mode across every test on this menu is running one that can't actually fail — where any plausible result gets rationalized as a pass. Watch for these tells:

• No threshold set before the test — "let's see the numbers and decide" invites post-hoc rationalization.
• Vanity traffic — driving friends, colleagues, or your own Twitter following instead of a representative audience inflates demand signals meaninglessly.
• Conflating curiosity with commitment — a click or an email signup costs the user nothing; only a real friction point (payment, calendar hold, waitlist deposit) tests actual intent.
• Moving the goalposts mid-test — deciding "actually, 4% would've been fine" after seeing 4.1%.

If you can't articulate, before launch, what result would make you kill the idea, you haven't designed a test — you've designed a confirmation exercise.

Pair a landing-page test with the same interview discipline you'd use elsewhere in discovery: when you follow up with signups, ask open, non-leading questions rather than questions that fish for the answer you want to hear. A landing page that converts well but generates only vague, hedged interest in a follow-up call is a weaker signal than a lower conversion rate paired with specific, urgent language about the problem.

Choosing the right test for your riskiest assumption

The fastest way to pick is to ask which risk, if wrong, ends the project — then match it to the cheapest test that can kill it. Running a concierge MVP to test pure demand is overkill; running a fake door test to validate a complex AI feature's output quality is underkill. Mismatch either direction wastes the cycle.

1. Demand is the biggest unknown, feature is simple to describe → fake door or landing page.
2. Demand is established, but you're unsure the automated experience will feel good → Wizard of Oz.
3. You're unsure anyone wants the outcome AND unsure what delivering it actually requires → concierge MVP.
4. You're testing multiple positioning angles, not just "want vs. don't want" → landing page with variants.

This mirrors how a well-run weekly discovery habit works: small, continuous, falsifiable checks beat one large validation event, because you catch a wrong assumption in week two instead of month four.

Logging the assumption before you test it

Every scrappy test above only produces a clean signal if you know, precisely, what belief it was built to prove or kill — and that belief is easy to forget once the test itself starts generating exciting numbers. Writing it down first, with the threshold, is what turns "the click rate was fine" into an actual verdict.

In Prodinja, a natural place to do this is a Journal entry — logging the assumption ("PMs will act on an automated risk digest"), the test design, and the pass/fail line before the test goes live, using Prodinja's real voice-capture Journals so the note is quick enough to actually write in the moment. When results come in, the same entry is where you record what happened against the threshold you set — so the verdict is anchored to the belief, not reconstructed from memory afterward.

Key Takeaways

• Match the test to the risk. Fake door and landing-page tests retire demand; Wizard of Oz and concierge MVP retire value and feasibility — they are not interchangeable.
• Set the pass/fail threshold before you launch, not after you see the numbers — this is the single biggest determinant of whether a test can actually falsify anything.
• A fake door test only proves curiosity, never value — a high click rate still leaves the core "will they actually use it" question open.
• Wizard of Oz hides the human; concierge discloses the human — pick based on whether the offer is algorithmic (hide it, then disclose before scaling) or service-shaped (disclose upfront).
• Vanity traffic and moved goalposts are the two most common ways a test becomes unfalsifiable — recruit representative users and refuse to renegotiate the bar after seeing results.
• Log the assumption, the test design, and the threshold together, before running the test — in a Journal entry or equivalent — so the result gets judged against a fixed bar, not rationalized after the fact.