Do You Even Have the Data? A Readiness Audit Before You Promise an AI Feature

You have enough data for an AI feature only when five separate things are true at once: enough volume for the task's actual complexity, labels tied to the real outcome, freshness that beats drift, verified access rights, and a sample that represents the users who'll see it. A feeling isn't an audit.

Quick Answer: Before you commit a ship date, run a five-point audit — volume vs. task complexity, label availability against the true outcome, freshness/drift, access and permissions, and representativeness. Fail any one of these, and your estimate is wrong before engineering writes a line of code.

Most AI feature timelines don't slip because the model underperforms. They slip because someone discovers, weeks into the build, that the data everyone assumed existed doesn't exist in the form the model needs. "We have tons of data" is a sentiment, not a specification — and it's the single most expensive sentence in AI product management, because it gets repeated in a roadmap review and then quietly falsified in a data-engineering standup six weeks later.

This audit exists to move that discovery earlier. It won't tell you whether your idea is good — only whether the thing you're about to promise is buildable with what you actually have.

Before you even get to data, decide whether the task needs machine learning at all with a clear-eyed look at the rules vs. ML vs. LLM decision. This piece assumes you've made that call and are scoping the data side of a feasibility review — the same one covered end-to-end in our complete guide to AI feasibility.

Volume vs. Task Complexity: How Much Data Is Actually Enough

"Enough data" isn't a fixed number of rows — it's a ratio between the signal you have and how hard the task is. A task with a clean decision boundary and a handful of features needs far less data than one with subtle, overlapping classes and dozens of interacting variables. Compare your volume against your task's complexity, not against how big the export file feels.

Three variables drive the real requirement, and none of them is "total rows in the table":

• Number of classes or outcomes. Predicting one of two outcomes needs less data per class than predicting one of twelve.
• Feature dimensionality. More input variables mean more combinations the model has to see examples of before it generalizes.
• Class balance. A rare-event task (fraud, safety incidents, high-value churn) is bottlenecked by how many positive examples exist, not by total record count.

These are order-of-magnitude starting points, not guarantees — treat them as a sanity check, not a spec. Google's internal Rules of Machine Learning guidance, written by engineer Martin Zinkevich, opens with a rule that sounds counterintuitive from a PM seat: don't be afraid to launch a product without machine learning at all. A simple heuristic baseline forces you to confront your actual data volume and quality before you've spent engineering time building infrastructure around a model that a spreadsheet formula could have approximated.

The practical move: before you scope the ML version, build (or ask an engineer to sketch) the dumbest rules-based version of the feature. If that version is also starved for signal, no model complexity will fix it — the problem is upstream of algorithm choice.

The Label Trap: Data That Exists but Isn't Labeled With the Outcome

Having raw behavioral data is not the same as having labeled data on the outcome you're trying to predict. For a churn-prediction feature, usage logs, support tickets, and billing history are not "churn data" until someone has tagged, for every account, whether and when it actually churned — and defined what "churned" means in the first place.

This is where "we have tons of data" quietly falls apart. A team scoping churn prediction typically does have enormous volume: months of login events, feature-usage telemetry, ticket histories. What it usually doesn't have is a clean, consistent label — a field that says "this account churned on this date, for this reason" — because nobody needed that field to exist until now.

Why the churn example changes the estimate

Consider a realistic version of this. A PM scopes a churn-prediction feature assuming a six-week build: two weeks of data prep, three weeks of modeling, one week of integration. The data audit turns up three problems that were invisible in the "we have tons of data" conversation:

1. "Churn" was never defined consistently. Product counts a cancelled subscription as churn; finance counts non-renewal at contract end; support counts 90 days of inactivity. These produce three different label sets from the same raw data.
2. The label has to be backfilled by hand or by heuristic, because no system stamped historical accounts with a churn outcome at the time it happened — it has to be reconstructed after the fact, which introduces its own error.
3. Label leakage risk is high. Support-ticket volume often spikes right before a churn event, which can make a model look artificially accurate by learning "customer complained a lot" instead of the earlier behavioral signal you actually want to act on.

None of that shows up until someone tries to build the label column. Resolving it — agreeing on a single churn definition with stakeholders, backfilling or approximating historical labels, auditing for leakage — is realistically its own three-to-four-week project before the original six-week model timeline even starts. The estimate doesn't change by a little; it roughly doubles, and the extra time is spent on data work, not modeling.

Andrew Ng's public case for data-centric AI makes the same point from the practitioner side: for most real-world projects, the highest-leverage work is improving the consistency and correctness of labels, not tuning model architecture. Teams that treat labeling as a quick data-prep step, rather than as the project's own workstream, are the ones who get the estimate wrong.

Getting the churn label right also means being precise about which customer outcome you're actually predicting — the same clarity that Jobs-to-Be-Done thinking forces onto feature scoping generally: churn isn't one event, it's the customer concluding your product no longer does the job they hired it for, and different jobs produce different churn signatures.

Ask these questions before you estimate anything:

• Does a field exist today that records the actual outcome, or will one have to be constructed after the fact?
• If multiple teams define the outcome differently, has anyone reconciled that into one definition?
• Were the labels assigned by a consistent process, or by whichever person happened to tag the account?
• Could any input feature only exist because the outcome already happened (leakage)?

Freshness and Drift: Why Last Year's Data Lies About Next Quarter

Data freshness matters because user behavior, product surface area, and market conditions change continuously — a model trained on eighteen-month-old data can backtest beautifully while being systematically wrong on live traffic today. Check how fast your domain moves before you promise an accuracy number.

Two distinct failure modes hide under the word "drift," and they call for different fixes:

• Data drift — the statistical properties of your inputs shift (a pricing change alters typical transaction size; a new onboarding flow changes early-usage patterns), even though the underlying relationship between inputs and outcome is unchanged.
• Concept drift — the relationship itself changes (the reasons customers churn six months after a category-wide price hike are different from the reasons they churned last year, even if the input features look similar).

A churn model trained entirely on data from before a pricing change, a competitor launch, or a major UI redesign is learning patterns from a world that no longer exists. It will often look accurate against a held-out test set drawn from the same stale period — that's the trap. Backtest accuracy on old data tells you almost nothing about live accuracy on new behavior once the underlying drivers have moved.

Freshness questions worth asking before you commit:

1. How far back does the training window need to reach to get enough volume, and has anything material changed in the product or market since then?
2. What's your plan for detecting drift once the feature ships — a monitoring metric, a scheduled retraining cadence, or nothing?
3. Is there a seasonal pattern (billing cycles, fiscal-year renewals, holiday usage dips) that a short training window would miss entirely?

Freshness isn't only a modeling concern — it interacts with delivery mechanics too. A feature that needs to reflect this week's behavior, not last quarter's, has real implications for how the pipeline is architected and how often it can retrain and redeploy, which is worth scoping alongside latency budgets for AI features rather than as an afterthought once the model is "done."

Access and Permissions: The Audit Nobody Runs Until Legal Asks

Data that's abundant, labeled, and fresh can still be unusable if you don't have documented rights to use it for this specific purpose. Check consent scope, contractual restrictions, data residency rules, and whether the data even lives in a system your team can query — before you put a date on the roadmap, not after legal flags it.

Purpose limitation is the quiet killer here. Data collected under a consent notice that says "to provide customer support" isn't automatically usable for "to train a predictive model," even if it's the exact same rows sitting in the exact same table. This is a core principle in privacy regulation generally (it's explicit in the EU's GDPR, for instance), and it's routinely the thing that stalls an AI feature after the data audit otherwise passed.

Each row in that table is a real conversation, and each one takes calendar time that has nothing to do with model complexity. A feature blocked on access isn't a data problem you can engineer around — it's a negotiation you have to schedule.

Don't skip the boring question either: does the team that will build this feature actually have query access to the system where the data lives, today, with the right scopes? "The data exists somewhere in the company" and "my engineer can SELECT from it this sprint" are very different facts, and the gap between them is a common, avoidable source of slipped estimates.

Representativeness: Whose Behavior Did You Actually Capture

A dataset can be large, labeled, fresh, and fully accessible and still be wrong, if it doesn't represent the population the feature will actually serve. Sampling skewed toward your best customers, a single geography, or an old version of the product produces a model that fails precisely on the users it hasn't seen before — often your newest or most at-risk ones.

Churn prediction is again a good stress test, because the most obvious source of training data — accounts that fully churned — carries a survivorship-style bias: it excludes the much larger set of customers who nearly churned, were saved by an intervention, or downgraded instead of cancelling outright. A model trained only on hard-cancel accounts learns a narrower pattern than the one your team actually needs to act on.

Common representativeness gaps to check for:

• Segment skew — is the training data dominated by your largest or longest-tenured accounts, undersampling newer or smaller ones?
• Geographic or channel skew — does the data reflect one region, plan tier, or acquisition channel disproportionately?
• Journey-stage gaps — does the data span the full arc from onboarding through renewal, or mostly the middle of the customer journey, missing early-stage signals that predict trouble before it's visible in usage data?
• Cold-start blindness — new users or new accounts, by definition, have thin histories; a model trained on established accounts may simply have nothing useful to say about them.

The U.S. NIST AI Risk Management Framework treats this as a named risk category — it explicitly calls out data representativeness as part of what makes a system "valid and reliable," not a nice-to-have refinement layered on afterward. Treat it the same way in your audit: representativeness is a pass/fail check, not a footnote you address if there's time.

Running the Audit Before You Commit a Date

Turn the five dimensions above into a single scorecard you fill out with engineering and data stakeholders before quoting a delivery date. Score each dimension green, yellow, or red; any red is a scope or timeline change, not a footnote in a status update.

A workable version of that scorecard:

1. Volume vs. complexity — do we have an order-of-magnitude estimate of examples needed for this task, and do we actually have that many?
2. Labels — does a field exist today recording the real outcome, defined consistently, assigned without leakage?
3. Freshness — has anything material changed since the training window started, and do we have a retraining plan?
4. Access — do we have documented purpose-appropriate rights, and does the build team have working query access right now?
5. Representativeness — does the sample span the segments, geographies, and journey stages the feature will actually serve?

Gartner analysts have repeatedly warned that a large share of AI initiatives fail to deliver reliable outcomes in production, and they point to data and bias problems — not model selection — as the dominant cause. That matches what shows up in practice: teams rarely get burned by picking the wrong algorithm. They get burned by skipping this audit and discovering the gap live.

This is also exactly the kind of question that's easy to wave past under deadline pressure, which is why Prodinja's Feature-to-Feasibility tool treats data readiness as its own explicit spec dimension rather than a side conversation. It's designed to walk you through naming the actual data source behind the ask, estimating volume against task complexity, and stating the labeling reality in writing — so "we have tons of data" gets replaced with a specific, checkable claim before feasibility gets scored.

A data audit is also only one input engineering needs before it can commit to a build. Pair it with the rest of the six questions engineering needs before it will write an AI spec — data readiness answers one of them well; the others (latency, failure modes, evaluation criteria) still need their own honest answers.

Key Takeaways

• "We have tons of data" is a feeling, not an audit — it describes volume, and says nothing about labels, freshness, access, or representativeness.
• Volume requirements scale with task complexity, not with row count — a rare-event or high-dimensional task needs far more examples than a simple binary classification with clean boundaries.
• The label trap is the most common estimate-killer: raw behavioral data (logs, tickets, billing) isn't a label until someone defines the outcome consistently and backfills or captures it without leakage.
• Freshness failures hide inside good backtest numbers — a model can look accurate on stale historical data while being systematically wrong on live traffic once drift has set in.
• Access and permissions are a scheduling problem, not an engineering one — purpose limitation, contractual restrictions, and residency rules take calendar time to resolve regardless of model readiness.
• Representativeness gaps fail silently on exactly the users you haven't captured yet — new accounts, underrepresented segments, or early journey stages.
• Score all five dimensions before quoting a date, and treat any red as a scope change communicated up front, not a surprise discovered mid-build.