An AI feature's maintenance bill is the recurring cost of detecting data drift and concept drift, running monitoring dashboards, and retraining models on a fixed cadence — commonly 15-30% of the original build cost, every year, for as long as the feature stays live. Skip the estimate and you inherit the bill anyway, just later, unbudgeted, and during an incident.

Quick answer: Budget an ongoing 15-30% of build cost per year for drift monitoring and retraining, name a maintenance owner before you ship, and treat "who watches this, and how will we know it's degrading?" as a launch gate — not a follow-up question.

Most AI feature specs price the build and stop there. The model ships, the demo works, everyone moves to the next roadmap item. Then, three to nine months later, someone notices the recommendations look stale, the fraud model is flagging the wrong transactions, or the support-ticket classifier has quietly started routing billing questions to the wrong queue — and there's no owner, no dashboard, and no budget line for fixing it.

That gap exists because an AI feature isn't a one-time build. It's closer to a subscription you pay in engineering time, for as long as the feature stays live, whether or not anyone budgeted for the renewal. This piece walks through what decays, what you need to watch, how often you'll need to retrain, and how to fold all three into a number you commit to before launch — not after the first incident. It's the natural sequel to a complete framework for assessing AI feasibility: feasibility gets you to launch; this is what it costs to stay launched.

Data Drift vs. Concept Drift: What's Actually Decaying

Two distinct failure modes cause an AI feature to degrade after launch. Data drift is when the inputs a model sees in production shift away from the distribution it was trained on. Concept drift is when the relationship between inputs and the correct output changes, even if the inputs look statistically identical. Both erode accuracy silently, and each needs a different fix.

Data Drift: The World Feeding Your Model Changes

Data drift shows up as a change in the inputs, not the underlying rule. A churn model trained on last year's customer mix starts seeing a new acquisition channel with different usage patterns. A pricing model trained pre-inflation starts seeing transaction amounts that no longer resemble its training range. The model itself hasn't gotten worse; the world it's scoring has moved out from under it.

This is exactly the failure mode a pre-launch data readiness audit for AI features is meant to catch early — except drift means the audit isn't a one-time gate, it's a recurring check against the same baseline.

Concept Drift: The Rules Your Model Learned Stop Holding

Concept drift is subtler and usually more damaging. The inputs can look completely normal while the mapping from input to correct output has changed. A fraud model trained on last year's attack patterns keeps seeing normal-looking transactions, but fraudsters have adapted their tactics — the label "fraudulent" now correlates with different features than it did at training time.

Researchers Gama, Žliobaitė, Bifet, Pechenizkiy, and Bouchachia, in their widely cited concept drift survey in ACM Computing Surveys, distinguish drift by shape — sudden, gradual, incremental, and recurring — because each shape demands a different detection window and response speed. A sudden drift (a policy change, a new competitor, a regulatory shift) needs near-real-time alerting; a gradual drift can be caught on a weekly or monthly review.

DimensionData DriftConcept Drift
What changesDistribution of input featuresRelationship between inputs and correct output
Typical causeNew user segment, new channel, seasonality, upstream schema changeBehavior change, adversarial adaptation, market/policy shift
Detection signalStatistical distance between training and live input distributions (e.g. PSI, KL divergence, KS test)Divergence between predicted and actual outcomes once ground truth arrives
Typical fixRetrain on recent data; re-check feature pipeline for schema driftRetrain with updated labels; sometimes requires new features entirely
Detection lagCan be near real-timeOften delayed until ground truth (a label) is observable

Both forms of drift are the practical, ongoing expression of the point Google researchers Sculley et al. made in their well-known paper on hidden technical debt in machine learning systems: production ML systems carry a CACE property — Changing Anything Changes Everything. A model isn't a static artifact you ship once; it's a live dependency on a data-generating process you don't control, which is exactly why it needs monitoring, not just a launch checklist.

The Monitoring Stack You Must Build Before Launch

Monitoring an AI feature means instrumenting four layers, not one: input distributions, prediction distributions, delayed ground-truth outcomes, and the business metric the feature was supposed to move. Skipping any layer means you can be blind to a full category of drift for months.

  1. Input monitoring — track the statistical distribution of the features going into the model in production versus training. Tools built for this, such as Evidently AI, WhyLabs, Arize, or Fiddler, compute distance metrics (PSI, KL divergence, population comparisons) on a schedule and alert past a threshold.
  2. Prediction monitoring — watch the distribution of the model's outputs, not just its inputs. A sudden shift in the share of "high risk" predictions, even with stable inputs, is often the first visible sign of drift before ground truth confirms it.
  3. Outcome / ground-truth monitoring — the slowest but most decisive layer. Once real outcomes arrive (a chargeback confirmed, a support ticket resolved, a churned customer), compare them against the original prediction to compute live accuracy, precision, and recall — not just at launch, but on a rolling window.
  4. Business-metric monitoring — the outcome a stakeholder actually cares about (deflection rate, approval rate, revenue per recommendation). This is the layer that turns a modeling problem into a business conversation, and it's the layer most teams forget to instrument at all.

None of this is free, and none of it is optional. It belongs in the same conversation as the six questions engineering needs answered before building an AI feature spec — monitoring requirements, alert thresholds, and who's paged are spec decisions, not afterthoughts bolted on post-launch. The same discipline applies to performance: just as you'd define a latency budget for an AI feature before shipping, define a drift budget — the maximum acceptable divergence before someone is required to act.

Treat monitoring like you'd treat test coverage: a feature without a monitoring plan hasn't actually shipped, it's just been deployed.

Retraining Cadence: How Often, and What It Costs Each Time

Retraining cadence should be set by how fast the feature's environment moves, not by convenience or the calendar. Three broad approaches exist — scheduled, trigger-based, and continuous — and each carries a materially different cost profile, so the choice belongs in the estimate, not in whoever happens to notice the model is stale.

CadenceHow it worksCost profileBest fit
Scheduled (weekly/monthly/quarterly)Retrain on a fixed calendar regardless of measured driftPredictable engineering time; risk of retraining too early (wasted effort) or too late (accuracy loss in between)Stable domains with slow-moving concepts (e.g. document classification on a settled taxonomy)
Trigger-basedRetrain when a monitored metric crosses a defined drift thresholdLower average cost, but unpredictable timing means on-call time and labeling can spike without warningDomains with irregular drift (fraud, pricing, anything adversarial)
Continuous / online learningModel updates incrementally as new labeled data arrivesHighest infrastructure and validation cost; needs strong guardrails against learning from a bad batchHigh-velocity domains where even weekly retraining is too slow (ad ranking, real-time bidding)

Each retraining cycle carries its own recurring costs, and they compound quietly:

  • Data labeling — fresh ground truth doesn't label itself; budget ongoing annotation or a human-in-the-loop review process.
  • Validation and rollback testing — a retrained model must be validated against held-out data and a rollback plan before it replaces the live model, every single cycle.
  • Compute — training runs, especially for larger models, are a recurring line item, not a one-time expense.
  • Human review — someone has to actually look at the drift report and decide whether to retrain, expand the feature set, or escalate; that's a person's time, every cycle, indefinitely.

A rough industry-level signal on why this matters: Gartner has repeatedly flagged that a majority of AI and machine learning initiatives stall or get abandoned after pilot, and unbudgeted operational cost — including exactly this kind of ongoing monitoring and retraining labor — is a recurring theme in that research, not a one-off finding. The build was never the hard part; sustaining it was.

Turning the Maintenance Bill Into a Real TCO Number

The maintenance bill becomes a defensible number the moment you treat it as a recurring percentage of build cost rather than a vague "we'll handle it later." A workable starting formula: annual maintenance cost ≈ monitoring infrastructure + on-call/review time + retraining cycles (compute + labeling + validation) + incident response, estimated across the feature's expected lifespan.

Directionally, teams that have gone through this exercise tend to land in a 15-30% of original build cost, per year range for anything beyond the simplest, most stable model — higher for adversarial domains like fraud or spam, lower for stable classification tasks with infrequent concept drift. That range is a planning anchor, not a guarantee; the point is to force the number to exist at all, before launch, instead of discovering it live.

This is also where it's worth reconnecting the feature to why it exists. A model that quietly degrades doesn't just lose accuracy points on a dashboard — it stops doing the job it was hired to do for the customer who depends on it. Mapping where that failure actually surfaces is easier with a complete guide to customer journey mapping in hand, because users rarely report "the model drifted" — they report a specific step in their journey that suddenly feels worse, wrong, or untrustworthy, and tracing that back to a drifting model is the harder diagnostic step.

Building the TCO estimate before commitment, not after, changes the decision itself. A feature that looks cheap at build time and expensive at year two isn't a bad feature — it's a feature whose real cost was hidden from the people who approved it. Surfacing the number earlier doesn't kill good ideas; it kills the ones that were never actually worth their ongoing bill.

The Gating Question: Who Watches This, and How Will We Know It's Degrading?

Every AI feature needs a named answer to one question before it ships: who watches this, and how will we know it's degrading? If nobody can answer both halves — a specific owner and a specific detection mechanism — the feature isn't ready to launch, regardless of how good the model looked in testing.

In practice, that means the spec needs to state, in plain language:

  1. Who owns it — a named person or team, not "the ML team" as an abstraction, responsible for reviewing drift reports on a defined cadence.
  2. What they're watching — which of the four monitoring layers above, at what threshold, triggers a look versus an alert versus a rollback.
  3. What happens when it degrades — a documented escalation path, not an assumption that someone will notice and improvise.
  4. When it gets reviewed regardless — a standing checkpoint (monthly, quarterly) even if no alert has fired, because silent, slow concept drift is exactly the kind that never crosses a threshold cleanly.

This is a scoping-time decision, not a post-launch cleanup task, and it's exactly the kind of thing that's easy to skip when a team is excited to ship. In Prodinja's prototype, the Feature-to-Feasibility tool is built around prompting for this directly — as part of scoping a feature, it walks you through naming a maintenance owner and describing the monitoring need, so the ongoing cost becomes part of the feasibility estimate itself rather than a surprise discovered during an incident review six months after launch. It's a structured nudge to answer the gating question before the roadmap commitment, not a system that diagnoses drift for you.

Key Takeaways

  • Data drift and concept drift are different failure modes — inputs shifting versus the input-output relationship changing — and each needs its own detection approach and response speed.
  • Monitoring needs four layers: input distributions, prediction distributions, delayed ground-truth outcomes, and the business metric the feature was meant to move.
  • Retraining cadence should match how fast the environment moves — scheduled, trigger-based, or continuous — and each has a materially different, recurring cost profile.
  • Budget 15-30% of build cost per year as a directional planning anchor for ongoing monitoring and retraining, higher in adversarial or fast-moving domains.
  • "Who watches this, and how will we know it's degrading?" is a launch gate, not a follow-up question — a feature without a named owner and detection plan isn't actually ready to ship.
  • The maintenance bill belongs in the original estimate, not discovered later — treating an AI feature as a recurring subscription in engineering time, rather than a one-time build, is what makes the total cost of ownership honest.

Frequently Asked Questions

How much should I budget for ongoing AI model maintenance?

A common directional planning range is 15-30% of the original build cost per year, covering monitoring infrastructure, periodic retraining, and the review time to interpret drift reports. Adversarial domains like fraud or spam detection tend toward the higher end; stable, low-drift classification tasks toward the lower end.

What's the difference between data drift and concept drift?

Data drift is a change in the inputs a model sees — a new user segment or shifted feature distribution — while the underlying input-output relationship stays the same. Concept drift is a change in that relationship itself, meaning the same-looking inputs now warrant a different correct output. Both degrade accuracy, but they need different detection signals and fixes.

How often should a machine learning model be retrained?

It depends on how fast the environment moves, not a fixed calendar: scheduled retraining (weekly, monthly, or quarterly) suits stable domains, trigger-based retraining suits environments with irregular or adversarial drift, and continuous/online learning suits high-velocity domains like real-time bidding. The right cadence should be chosen deliberately and priced into the estimate, not defaulted to "whenever someone notices."

Do all AI features need dedicated drift monitoring?

Any AI feature making decisions that affect users or revenue over time needs at least input and outcome monitoring, even if lightweight. A low-stakes, rarely-used feature might justify a simple quarterly manual check; a high-stakes, high-volume feature (fraud, pricing, recommendations) needs the full four-layer stack described above, instrumented before launch, not after a complaint.

Who should own monitoring an AI feature after launch?

A named individual or team, stated in the spec before launch — never left as an implicit assumption that "the ML team will handle it." The owner should have a defined review cadence, clear alert thresholds, and an escalation path, so degradation surfaces as a scheduled check or an alert rather than a customer complaint.