Working With Data Scientists: Why Their Experiments Aren't Your Features (Yet)

Data scientists and PMs fail each other constantly because they define success differently. An experiment succeeds when it produces a trustworthy answer — even a negative one. A feature succeeds only when it ships and creates value. Treat the two as interchangeable line items on a roadmap, and you'll ship broken promises instead of working software.

Quick answer: An ML experiment is a bet on learning something; a feature is a bet on delivering value. Estimate them on separate tracks — an exploration budget for the former, a delivery commitment for the latter — and tell leadership which track each date belongs to.

An Experiment Can "Succeed" by Proving You Wrong

In science, a well-run experiment succeeds when it produces a reliable, well-measured answer, regardless of what that answer is. In product work, a feature succeeds only when it ships and improves an outcome for users or the business. Confusing these two definitions is the single biggest source of friction in any AI PM data science collaboration.

This isn't a semantic nitpick. It changes how you plan, how you report status, and how you protect your data scientists from being punished for doing their jobs correctly.

• Data scientist's success: a hypothesis gets tested rigorously enough that the team can act on the answer, whatever it turns out to be.
• PM's default success (learned from feature work): a scoped deliverable ships on a date and moves a business metric.
• Where they collide: a roadmap slide lists "Recommendation engine v2" next to a date, as if it were a known deliverable instead of an open question with a coin-flip outcome.

Ron Kohavi, who ran the experimentation platforms at Microsoft and Airbnb and co-wrote Trustworthy Online Controlled Experiments, has published one of the more sobering numbers in this space: across large, mature testing programs, only around a third of ideas that get tested actually improve the metric they were built to move. Most either do nothing measurable or make things worse. If a third is a good hit rate at Microsoft's scale, treating every experiment on your roadmap as a guaranteed win is not optimism — it's a planning error.

The practical fix starts with vocabulary. Stop calling a hypothesis test a "deliverable." Start calling it what it is: a question with a deadline attached to the answer, not to a working feature.

The Research-to-Production Funnel (Where PM Estimates Break)

Every ML idea moves through distinct stages — research question, offline prototype, offline evaluation, online experiment, and productionization — and each stage carries a different failure rate and a different kind of uncertainty. PMs who estimate the whole funnel like a single feature ticket are actually estimating five different problems as if they were one.

Here's what tends to happen at each stage, and the gap between what a PM assumes and what's actually true:

That last row isn't a guess. In "Hidden Technical Debt in Machine Learning Systems," Google researcher D. Sculley and co-authors showed that in real-world ML systems, the model code itself is often a small sliver of the total system — surrounded by far larger amounts of data verification, feature plumbing, monitoring, and configuration. The "just wire it up" stage is frequently the most expensive one, not the cheapest.

The funnel also explains why so few experiments ever become features at all. Surveys of data science teams compiled by outlets like VentureBeat have repeatedly found that a large majority of models built inside companies — often cited near 87 percent — never reach production. That's not a story about incompetent teams. It's the natural shape of a funnel where most ideas are supposed to get filtered out before they cost real engineering time.

For PMs coming from a more traditional analytics background, none of this should feel new; before an experiment can even start, the underlying data has to be reliable enough to trust, which is exactly the kind of pipeline and quality work covered in our guide to the data PM role. What's different here is that even with clean data, the outcome of the analysis stays genuinely unknown until you run it.

Why Estimating Experiments Like Features Poisons Trust

When a PM puts a ship date on a hypothesis test, one of two bad things happens: the team quietly extends the experiment until it produces the "right" answer, or leadership loses trust the first time an honest null result gets reported as a miss. Both outcomes erode the one thing you actually need from a research partner — candor.

The quiet extension is the more dangerous failure mode because it's invisible until much later. A data scientist under pressure to hit a date will keep slicing the data until something looks significant — a version of what statisticians call p-hacking. You get a "positive" result that doesn't replicate, ship a feature on top of it, and watch the metric fail to move six weeks after launch. Now you've spent delivery-team calendar time on a result that was never real.

Amazon founder Jeff Bezos drew a durable distinction in his 2015 shareholder letter: one-way-door decisions are hard to reverse and deserve heavy deliberation; two-way-door decisions are cheap to reverse and should be made fast, without waiting for full consensus. Most ML experiments are two-way doors — you can kill them cheaply and walk away with only the sunk analyst-hours as cost. Shipping a feature to every user, deprecating an old system, or putting a launch date in front of leadership are one-way doors. The mistake is running a two-way-door activity with the ceremony, sign-off, and calendar weight of a one-way door.

Growth PMs already have useful instincts here. Running dozens of small, cheap, reversible tests a quarter — the day-to-day rhythm covered in our guide to the growth PM role — builds exactly the muscle of treating an experiment as a two-way door. The gap is that ML experiments often have longer cycle times and higher setup cost than a landing-page test, so the same "just ship the test and see" instinct needs a heavier kill-criteria discipline layered on top, or it produces estimate whiplash instead of speed.

Platform PMs face a related version of this problem from the infrastructure side. Retraining pipelines, feature stores, and deprecating an old scoring service are the kind of one-way-door infrastructure calls examined in our guide to the platform PM role — and they're exactly where a "quick experiment" quietly turns into a permanent dependency nobody budgeted for.

A Framework: Exploration Budget vs. Delivery Commitment

Split your team's capacity into two explicitly labeled buckets: an exploration budget, sized in time or headcount rather than outcomes, that funds hypothesis-testing with no promised result; and a delivery commitment, sized in scope and date, that only starts once an idea has cleared explicit graduation criteria out of the first bucket.

The two buckets need different vocabulary, different owners, and different reporting formats, or people will quietly start borrowing the language of one for the other.

Putting this in place is less about a new tool and more about five habits:

1. Ring-fence a real percentage of capacity as exploration budget. Google's old "20% time" is a rough cultural analog, not a literal template — the point is that the allocation is named, visible, and defended, not scraped from whatever's left over after delivery work.
2. Pre-register the hypothesis and the kill criteria before analysis starts. This mirrors the assumption-mapping discipline Teresa Torres teaches in Continuous Discovery Habits — decide what evidence would change your mind before you're emotionally invested in a specific answer.
3. Report exploration in decisions-per-period, not features-per-period. "We resolved four open questions this quarter" is a legitimate, measurable output even if zero of them shipped as features.
4. Define explicit graduation criteria for moving an idea from exploration into a delivery commitment — a minimum effect size, a confidence threshold, an engineering feasibility check.
5. Require a new, explicit request to extend a time-boxed experiment. Silent drift past the original deadline is how exploration budget quietly eats delivery-commitment calendar time.

The RICE and Kano techniques you might already use for feature prioritization are built for the delivery side of this table — they assume you already know roughly what a feature will do. Trying to run them on an unproven hypothesis just launders uncertainty into a false-precision score.

The Script: Communicating Experiment Risk to Leadership

Leadership doesn't need to understand p-values; they need three things at every check-in: what decision this experiment will produce, by when, and what happens under each possible outcome. Give them that in plain language, and a null result stops looking like a failure and starts looking like exactly what they paid for.

Use three short, repeatable messages instead of one dense status update. Each one takes under a minute to say out loud.

Before You Start: The Framing Message

"We're spending two data-scientist-weeks to answer whether personalized ranking improves session length. This is exploration budget, not a delivery commitment — there's no feature attached to a positive result yet. We'll know by the 14th whether to scale it, run a follow-up test, or close it out."

The Midpoint Check-In

"Halfway through the test. Early signal is inconclusive — sample size isn't there yet. No decision needed from you today; flagging now so the 14th isn't a surprise either way."

The Readout (Whatever the Result)

"The experiment answered the question: personalized ranking did not move session length outside the noise band. Here's what we now know that we didn't two weeks ago, and here's the recommendation — close this direction out and redirect the budget to the onboarding-friction hypothesis instead."

Notice what's missing from all three: an apology. If the exploration budget was framed correctly at kickoff, a negative or inconclusive result requires no defense — it requires a recommendation. This translation work, turning statistical uncertainty into a plain business decision, is a big part of what makes the roles in our guide to the technical PM role distinct from more purely delivery-focused product roles.

The PM's Real Job in This Relationship

The PM's job here isn't to make data scientists estimate better — it's to build the organizational plumbing that lets exploration and delivery coexist without either side lying to leadership about what it is. That plumbing is process, not persuasion.

Working with data scientists as a PM means holding two different mental models at once: comfortable with sprint mechanics and readiness gates, and equally comfortable with confidence intervals and kill criteria. That dual fluency is a big part of what separates the emerging AI PM role from a traditional software PM role, a distinction we go deeper on in our comparison of AI PM and software PM responsibilities. Get the framing wrong, and the tension of ML experiments vs. features shows up first on the calendar — long before it shows up in the model.

Where the Learning Goes When the Experiment Doesn't "Work"

Most inconclusive experiments die twice: once when the metric doesn't move, and again when the reasoning behind it evaporates from someone's memory or a stale Slack thread. The fix is treating every experiment cycle as a logged assumption and a logged friction, not just a pass/fail outcome.

This is the specific gap Prodinja's Journals are built to close in its current prototype. Instead of relying on someone to write up meeting notes after the fact, Journals let you log Assumptions and Frictions as they surface during an experiment cycle, with real voice capture — so a five-minute debrief after a null result becomes a searchable, dated entry instead of a lost hallway conversation. It doesn't run the experiment or interpret the statistics for you; it's designed to make sure the decision and the reasoning behind it survive past the person who made them, which is often the real reason "didn't we already try that?" conversations keep recurring a year later.

None of this replaces rigor at the analysis stage. It just means the exploration budget you spent buys a durable asset — a dated, attributable record of what was tried and why — even when the headline result is "no."

Key Takeaways

• An experiment succeeds by producing a trustworthy answer; a feature succeeds by shipping value — don't collapse the two definitions into one roadmap line.
• Estimate the research-to-production funnel stage by stage; research question, offline evaluation, online experiment, and productionization each carry a different failure rate.
• Split team capacity into an explicit exploration budget (time-boxed, no promised outcome) and a delivery commitment (scoped, dated) — and don't let either borrow the other's vocabulary.
• Use the one-way-door/two-way-door lens: most experiments are reversible bets, while features and public commitments are not, and each deserves a different level of ceremony.
• Give leadership a plain-language script at kickoff, midpoint, and readout so a null result reads as "we made a decision," not "we missed a date."
• Pre-register hypotheses and kill criteria before starting, the same discipline Teresa Torres recommends for testing assumptions in continuous discovery.
• Log assumptions and frictions as they happen instead of after the fact, so an inconclusive experiment still leaves the team smarter next quarter.

Frequently Asked Questions

How do I estimate a data science project if I can't predict the outcome?

You don't estimate the outcome — you estimate the time-box to reach a decision. Commit to a date by which you'll know whether to scale, iterate, or kill the idea, and treat that decision date as the deliverable instead of a specific result.

What percentage of ML experiments actually make it to production?

There's no universal number, but the directional industry data is consistently sobering. Surveys compiled by outlets like VentureBeat have found the large majority of data science models — often cited near 87 percent — never reach production, and Ron Kohavi's controlled-experiment research found only around a third of tested ideas move the metric they targeted in the intended direction.

How should a PM report a failed or inconclusive experiment to leadership?

Report it as a completed decision, not a missed deliverable: state the hypothesis, what was learned, and the recommendation — kill, iterate, or scale — in the same breath. If the exploration budget was framed correctly at kickoff, this update requires no defense.

Is working with data scientists really different from working with software engineers?

Yes — the core difference is what a ticket represents. A software engineering ticket is usually a known solution with an unknown implementation detail; a data science ticket is often an unknown solution with an unknown outcome, which is why sprint-style estimation translates so poorly from one to the other.

Should PMs sit in on data science experiment design?

Yes, at the framing stage — not to review the statistics, but to make sure the hypothesis, success metric, and kill criteria get decided before analysis starts, since these are exactly the inputs leadership will judge later regardless of what the result turns out to be.