Evals Don't End at Launch: Monitoring AI Quality in Production

Offline evals tell you whether your AI feature was good enough to ship. They say nothing about whether it still is next month. Production AI monitoring closes that gap: sampling live traffic through an LLM judge, tracking feedback and error rates, watching for input drift, and alerting before decay becomes a support ticket flood.

Quick Answer: Continuous evaluation in production means scoring a sample of real traffic against your eval rubric, tracking refusal/error rates, cost, latency, and input drift, and routing failures back into your golden dataset — all on a fixed cadence with someone accountable for reading the results.

Why Offline Evals Expire the Moment You Ship

Your offline eval suite is a snapshot of what you expected users to ask, scored against a golden set you built before launch. Production traffic doesn't stay inside that snapshot. Within weeks, real usage drifts far enough from your test cases that a fully green offline suite can sit right next to a visibly degrading live experience.

This isn't a one-time gap you can patch and forget. It's structural, and it recurs continuously:

• New user segments show up with phrasing, jargon, or intents your golden set never anticipated.
• Prompt injection and adversarial inputs appear the moment your feature is public, not before.
• Upstream model versions change silently — a provider ships a quiet update and your carefully tuned prompt behaves differently overnight.
• Downstream integrations shift, feeding your model different context than it saw in testing.
• Seasonality and product changes elsewhere in your app alter what people ask for.

The complete guide to AI evals frames offline evaluation as the gate you pass before launch. Production monitoring is the discipline that starts the moment that gate opens — and it never fully closes again. If your golden dataset is the ruler you measured launch-readiness with, treat it as a living document, not a launch artifact you file away.

The Five Signals Every Production AI Feature Should Track

Live monitoring boils down to five signal categories, each catching a different failure mode a golden-set eval can't. Miss any one and you get blindsided by a different kind of decay — a quiet quality slide, an angry user, a cost spiral, or a traffic shape you never tested against.

These five signals are widely recognized, not an idiosyncratic list. ML observability vendors such as Arize AI and WhyLabs have built entire product categories around this exact combination — drift detection, feedback capture, live scoring — a sign it's settled operating practice, not a nice-to-have.

OpenAI's open-source evals framework and Anthropic's own published guidance on evaluating AI systems both treat evaluation the same way: a continuous practice that follows the model into production, not a pre-launch checkbox you tick once.

Quality Scores and Feedback Are Not the Same Signal

A rising judge score and a rising thumbs-down rate can coexist, and when they do, it usually means your rubric is missing something users actually care about. Judges measure conformance to a spec; users measure whether the answer helped.

Track both, and treat a divergence between them as a prompt to revisit the rubric itself, not just the model.

Input Drift Is Often a Jobs Signal, Not a Data Signal

When the shape of incoming requests shifts, it's tempting to treat it as a pure statistics problem — a new cluster in your embedding space, a KL divergence reading that's crept upward. Often the more useful read is behavioral: users have started hiring your feature for a different job than the one you designed and tested for.

Mapping what changed against the jobs users are actually hiring your product to do turns a drift alert into a concrete hypothesis worth testing, rather than a number that just sits on a dashboard.

How to Sample Production Traffic With an LLM Judge

You can't have a human read every production response, so route a meaningful sample through an LLM judge using the same scoring rubric your offline evals use, logged continuously as a rolling quality metric. The trick is sampling smart, not sampling everything — random coverage plus targeted capture of the interactions most likely to be broken.

A workable pipeline looks like this:

1. Log every request and response with metadata: user segment, feature flag state, model version, latency, and any confidence signal your system exposes.
2. Sample in layers, not just randomly — a fixed random percentage for baseline trend, plus every low-confidence response, every response tied to negative feedback, and any flagged keyword or safety trigger.
3. Score with the same judge rubric as your offline suite, so a production number and an offline number mean the same thing and can be compared directly.
4. Spot-check the judge itself with periodic human review — a "judge of judges" pass that catches a judge quietly drifting off-rubric.
5. Aggregate into a rolling window (daily or weekly) rather than reacting to any single scored response.

No single sampling strategy covers everything on its own, which is why most mature setups run more than one at once:

Deciding how much to trust a judge's verdict, and when a disagreement between judge and human is the judge's fault rather than a genuinely ambiguous case, is its own skill. When to trust an LLM-as-judge is worth reading before you wire a judge into a live pipeline, not after you notice it's been quietly wrong for a month.

Eugene Yan's applied-ML writing on LLM evaluators is a useful independent check here too. His analysis of judge-human agreement rates is a good reminder that a judge is a calibrated instrument, not ground truth, and needs periodic recalibration like any other one.

Alerting Thresholds That Catch Real Decay Without Crying Wolf

Good production alerts combine a hard floor that should never be crossed with a relative threshold measured against a rolling baseline, and require the breach to persist across more than one measurement window before firing. A single bad batch of five requests should not page anyone at 2 a.m.

Four threshold types cover most of what you need, and they layer rather than compete:

• Hard floor/ceiling — a line that should never be crossed regardless of trend, e.g., refusal rate must never exceed a fixed percentage.
• Rolling baseline deviation — flag a score that falls meaningfully below its own recent history, e.g., more than two standard deviations below a 14-day rolling average.
• Sustained-duration gate — require the breach to hold across several consecutive windows before it escalates, so noise from one bad sample doesn't trigger a false alarm.
• Rate-of-change — catch a fast-moving trend, like a week-over-week jump in regeneration rate, before it crosses any absolute line at all.

The numeric-threshold discipline that makes an offline eval genuinely failable — a specific number, not a vague "seems fine" — applies directly here. A production alert with no numeric trigger is just a dashboard nobody reads until something is already on fire.

Google's Site Reliability Engineering practice popularized pairing every alert with an error budget: a threshold for how much degradation is tolerable before it becomes an incident, rather than treating every fluctuation as equally urgent. Borrowing that framing for AI quality metrics — deciding in advance how much judge-score dip or feedback decline is "within budget" — keeps a team from either ignoring real decay or drowning in false alarms.

Closing the Loop: Turning Production Failures into Golden Set Examples

Monitoring only pays off if what it finds flows back into your offline eval suite. Every low-scoring, negatively-flagged, or human-escalated production interaction is a candidate golden-set example — and the ones that are genuinely instructive, not just noisy one-offs, should get added so the next model or prompt version is tested against real failures you've actually seen.

This loop has a shape worth running deliberately rather than ad hoc:

1. Flag — anything with a low judge score, a negative user reaction, or a human support escalation gets tagged automatically.
2. Triage on a fixed cadence — weekly is common — sorting flagged items into genuine failures, ambiguous edge cases, and noise.
3. Root-cause tag each genuine failure (prompt gap, retrieval miss, drift, model regression) so patterns become visible over time.
4. Add representative cases — not all of them — to the golden set, favoring diversity of failure mode over sheer volume.
5. Re-run the offline eval suite against the updated set to confirm the next release is actually being tested against what broke last time.

Chip Huyen's writing on production ML systems describes this same feedback loop as the difference between a model that's evaluated once and a system that's continuously monitored — the golden set is never "done," because production keeps generating the examples that expose what it was missing.

A golden set that stops growing after launch is testing your feature against last quarter's users, not this quarter's.

Skipping this feedback step is a common reason teams re-discover the same failure mode release after release.

It's also worth noticing where these failures cluster relative to how someone actually uses your product. A failure that shows up right as a user is trying to complete a high-stakes step reads very differently from one buried in casual exploration. Overlaying flagged failures on a customer journey emotion curve tends to reveal that quality erosion concentrates at one or two specific moments rather than spreading evenly — which tells you where to prioritize the fix.

Building a Monitoring Cadence You'll Actually Keep

A monitoring setup with no owner and no schedule decays exactly like an unmaintained golden set — dashboards get built, then quietly stop getting opened. The fix isn't more tooling; it's a calendar-anchored review cadence with a named owner, the same discipline that keeps any recurring operational practice alive past its first enthusiastic month.

A workable cadence, roughly:

• Weekly — review sampled quality scores, feedback themes, and any triggered alerts from the past week.
• Biweekly — look at input distribution drift and whether new segments or use cases have emerged.
• Monthly — recalibrate thresholds against the last month's actual baseline, since a threshold set at launch is already stale a quarter in.
• Quarterly — a deeper golden-set refresh pulling the quarter's most instructive production failures.

Prodinja's own workflow reflects this split honestly. The Evals critique inside Spec Studio is built to walk you through drafting an offline eval plan before you ship — a golden set, numeric thresholds, and a judge rubric — as part of getting a feature ready for launch. What happens after launch is a different kind of discipline that no planning tool can automate for you: someone actually has to open the dashboards on schedule.

That's where Prodinja's Reminders feature is useful, in a narrower and more honest way. It's a real, working part of the prototype for setting recurring tasks — you can point one at "review production quality signals" on a weekly cycle, so the review doesn't quietly stop happening the way ad hoc habits tend to a few weeks after launch enthusiasm fades.

Key Takeaways

• Offline evals are a snapshot, not a guarantee — production traffic drifts from your golden set continuously, not just once at launch.
• Track five signal types together: sampled quality scores, user feedback, refusal/error rates, cost and latency, and input distribution drift.
• Layer your sampling strategy — random for baseline trend, stratified for segment coverage, targeted for fast failure discovery.
• Score production traffic with the same judge rubric as your offline evals, so the two numbers are directly comparable, and spot-check the judge itself periodically.
• Combine hard floors, rolling-baseline deviation, and a sustained-duration gate in your alerts so you catch real decay without paging on noise.
• Every genuine production failure is a candidate golden-set example — the loop back into your offline suite is what keeps evals relevant release after release.
• A monitoring cadence needs an owner and a schedule, or the dashboards quietly stop being read within a month of launch.