Offline evals answer one question: is this output correct, safe, or well-formed against a reference you already trust? Online evals answer another: did a real person behave differently because of it — click, convert, stay, or churn? A model can top your golden-set leaderboard and still move nothing in production, because these are two separate measurements of two separate things.

Quick answer: Offline evals score a model against a fixed, labeled dataset before launch — repeatable, cheap, and diagnostic. Online evals measure real user behavior after launch — noisy, expensive, and the only numbers that actually move the business. Treat every offline metric as a hypothesis about an online outcome, then check that hypothesis after you ship.

The Two Questions Hiding Inside Every Eval Metric

Offline evaluation asks whether a specific output is correct, safe, or well-formed against a reference you already trust. Online evaluation asks whether that output changed what a real person did next. Every metric you'll ever compute in an AI evals program belongs to one camp or the other, and conflating them is the most common design mistake teams make.

The confusion is understandable — both produce a number, both get called "the eval," and both show up on the same dashboard. But they answer to different owners: offline evals answer to the model, online evals answer to the business.

What Counts as an Offline Eval

Offline evals run against a golden dataset — a curated, labeled set of inputs and expected outputs that doesn't change between runs. You compute them in CI, before a model or prompt change ships, and you can rerun them a thousand times and get the same score.

Common offline metrics include:

  • Exact-match or F1 against known-correct answers
  • Faithfulness/groundedness scores for RAG outputs, checking claims against retrieved context
  • LLM-as-judge rubric scores for helpfulness, tone, or format compliance
  • Safety and toxicity classifier pass rates on adversarial prompts
  • Latency and cost benchmarks under simulated load

If you haven't built one yet, start small — a well-constructed 50-example golden dataset surfaces most of the failure modes a 500-example one will, just faster. And any judge you use to score these examples is only as trustworthy as its calibration against human raters — worth reading up on when to trust an LLM-as-judge before you lean on one for a launch decision.

This part of the discipline is standardized enough now that tooling exists for it: OpenAI open-sourced its evals framework in 2023 specifically to make golden-set testing a repeatable, version-controlled step rather than a one-off spreadsheet exercise. That's the right instinct — the mechanics of offline scoring are largely solved. What's much less standardized is the second half of this article: knowing which offline number to trust as a stand-in for real behavior.

What Counts as an Online Eval

Online evals run against live traffic, after launch, using an A/B test, holdout group, or interleaving design. There's no fixed answer key — the "ground truth" is whatever real users actually did.

Typical online metrics fall into a few families, close to what Google's HEART framework (Happiness, Engagement, Adoption, Retention, Task success) was built to organize back in 2010 for large-scale web UX measurement:

  • Task success — did the user complete what they came to do?
  • Engagement — session length, follow-up queries, repeat use
  • Retention — did they come back next week?
  • Satisfaction — CSAT, thumbs-up rate, NPS
  • Business outcomes — conversion, support ticket volume, revenue per session

Online evals are slower, noisier, and require real user volume to reach statistical significance. That's exactly why you don't want them to be the first time you learn something is broken.

Why This Split Matters More for AI Products Than Traditional Software

Traditional feature A/B tests compare two fixed, deterministic experiences — the offline-online gap mostly comes down to whether users like the new design. Generative AI adds a second layer of unpredictability: the model's output itself is non-deterministic, and it can drift underneath you when a vendor quietly updates a base model.

That means an AI product's online metrics can move for reasons that have nothing to do with the change you shipped — a provider-side model update, a shift in the prompts users happen to send that week, or seasonal changes in query mix. Guardrail metrics and a stable offline baseline are how you tell your change apart from that background noise.

Offline vs. Online Evaluation at a Glance

DimensionOffline EvaluationOnline Evaluation
Runs againstA fixed, labeled golden datasetLive production traffic
TimingPre-launch, in CI, on every changePost-launch, continuously
RepeatabilityDeterministic — same input, same scoreNon-repeatable — depends on real users that day
Speed of feedbackMinutes to hoursDays to weeks, until significance is reached
Typical ownerML/eval engineer, PM reviewing a launch gateGrowth, data science, product analytics
Answers"Is this output good?""Did users behave differently because of it?"
Primary riskDoesn't reflect real-world distributionConfounded by noise, seasonality, novelty effects

The Correlation Gap: When Offline Scores and Online Impact Diverge

The correlation gap is what happens when a model's offline score improves but the online metric it was supposed to move stays flat, or gets worse. It shows up constantly in recommendation systems, RAG assistants, and chat products, and it's the reason teams that only watch offline numbers get blindsided at launch.

This isn't a rare edge case — it's close to the median outcome. In his widely cited account of Microsoft's online experimentation platform, researcher Ron Kohavi has described a rough one-third split among tested product ideas: about a third move the target metric in the intended direction, a third show no meaningful effect, and a third actually make it worse. Most changes that look good on paper simply don't confirm in live behavior.

Netflix's recommendations team has written about a related version of this problem: offline ranking metrics like NDCG frequently fail to predict the online engagement lift measured in A/B tests, which is part of why the team leans on interleaving — blending two ranked lists in front of real users — to get a faster, cheaper read on which ranking actually wins before committing to a full experiment.

Why the Gap Opens Up

A handful of mechanisms explain most of the drift between a strong offline number and a flat online one:

  1. Distribution shift — your golden set was curated last quarter; live traffic already looks different.
  2. Invisible UX friction — latency, formatting, and tone affect trust and completion in ways a static reference answer can't capture.
  3. Judge bias without user bias — an LLM-as-judge or human rater rewards a style real users don't actually prefer.
  4. Novelty and habituation — a change that delights on day one gets ignored by day thirty, or vice versa.
  5. Goodhart's law in action — once a proxy metric becomes the target teams optimize for, it stops tracking the thing it was a proxy for.

None of these mean offline evals are useless — they mean offline evals are a filter, not a verdict. They should stop obviously broken models from reaching users; they were never going to tell you whether users will love what's left.

Machine learning engineer and consultant Hamel Husain, whose widely shared essay "Your AI Product Needs Evals" helped popularize disciplined eval practice among LLM product teams, makes a related point: teams often build elaborate offline scoring systems while skipping the harder, more valuable step of looking at actual production traces to see how the model is failing in the wild. The offline number without the online read is a plausible story, not a confirmed one.

Mapping Offline Metrics to the Online Outcomes They Predict

Every offline metric should exist because it's a leading indicator for a specific online outcome, never because it happens to be easy to compute. The table below is a starting map — swap the right-hand column for whatever your team actually tracks in production.

Offline Metric (pre-launch, on the golden set)Online Metric It's Meant to Predict
Exact-match / accuracy against known answersTask completion rate
faithfulness / groundedness score (RAG claims vs. retrieved context)Correction requests / support escalations
LLM-as-judge helpfulness score (rubric-based)Thumbs-up rate / CSAT
Retrieval precision@k (relevance of retrieved context)Follow-up query rate / session depth
Safety or toxicity classifier pass rateComplaint or escalation rate
Latency benchmark (P95, simulated load)Session abandonment rate
Coherence / fluency scoreTime-on-task / rephrase rate
Refusal-rate on an adversarial setRepeat-usage rate / 30-day churn

Read the table as a set of hypotheses, not facts. A high faithfulness score is supposed to reduce support escalations — but until you've measured escalation rate before and after a faithfulness-improving change, that's an assumption, not a result. The value of the table is that it forces you to name the assumption before you ship.

Common Mistakes When Building This Map

  • Picking the online metric after the fact, so it conveniently confirms whatever the offline score already showed.
  • Using one offline metric to justify multiple, unrelated online claims — a coherence score can't also stand in for trust and for task completion.
  • Never revisiting the pairing once it's documented, even as the product and its user base change shape.
  • Treating a directional correlation from one launch as permanent proof, rather than re-testing it on the next model or prompt change.

Building the Bridge: Designing Evals That Actually Predict Behavior

Bridging the gap means designing your offline plan backward from the online metric you actually care about, then verifying the link empirically instead of assuming it holds. Five practices consistently close the correlation gap in mature eval programs.

  1. Start from the online metric, not the offline one. Decide what job the user is actually hiring the product to do — the jobs-to-be-done framework is a sharper tool for this than a feature list — and only then pick offline metrics that plausibly predict it.
  2. Set numeric pass/fail thresholds before launch, not after. A vague "looks better" isn't a gate; failable evals with explicit numeric thresholds are what actually stop a regression from shipping.
  3. Run shadow or canary traffic before a full rollout. Route a small slice of real traffic through the new model without showing it to users, or expose it to a small cohort first, so your first online read comes from real behavior, not a launch-day surprise.
  4. Track guardrail metrics alongside your primary one. Kohavi's experimentation work popularized the idea of an Overall Evaluation Criterion paired with guardrails — you don't just want the target metric up, you want to confirm nothing else quietly broke.
  5. Re-validate the offline-online link on a schedule. A golden set built six months ago reflects six-month-old usage patterns; user behavior — and the customer journey stage they're in when they hit your product — shifts underneath it.

A Note on LLM-as-Judge Scores Specifically

LLM-as-judge scores deserve extra scrutiny in this bridge-building process, because they're the offline metric most likely to feel like ground truth without actually being it. A judge model can reliably rank outputs by a rubric while still being poorly calibrated to what real users find helpful — which is exactly why judge scores need periodic correlation checks against actual online satisfaction data, not a one-time validation.

Statistical Rigor: Don't Let Noise Masquerade as a Result

An online metric that "moved" isn't automatically a signal — it might just be noise. Ron Kohavi, along with co-authors Diane Tang and Ya Xu, devoted much of their book Trustworthy Online Controlled Experiments to exactly this trap: teams routinely under-power a test, then treat a random fluctuation as proof a change worked.

Before you launch an online eval, work out the minimum sample size your primary metric needs to detect the effect size you actually care about. If your product doesn't have that traffic volume yet, a holdout comparison over a longer time window, or a qualitative signal alongside the quantitative one, is more honest than declaring victory on an underpowered test.

Where This Fits in Your Evaluation Strategy

A real evaluation strategy is just this offline-online link written down and revisited on purpose, rather than assumed. Know what "good" means before launch, and know exactly which live metric that offline bar is supposed to predict — that sequence, not either half alone, is what an evaluation strategy actually is.

That's the sequence Prodinja's Evals tool is designed around: it walks you through constructing the offline plan — the golden set, the judge rubric, the numeric pass/fail thresholds — that has to clear before a change ships. Because you've already named what "good" means offline, that same plan naturally sets up the online metrics worth watching once real users are in the loop, instead of leaving the mapping as an afterthought you improvise post-launch.

Key Takeaways

  • Offline evals answer "is the model good?" against a fixed, labeled dataset — repeatable, cheap, and run before launch.
  • Online evals answer "do users behave better?" using live A/B tests or holdouts — noisier, slower, and the only numbers tied to business impact.
  • The correlation gap is common, not rare — Ron Kohavi's Microsoft data suggests roughly a third of tested ideas fail to move their target metric at all.
  • Every offline metric should map to a specific online outcome it's meant to predict — treat the mapping as a hypothesis, not a guarantee.
  • Distribution shift, invisible UX friction, and Goodhart's law are the usual reasons a strong offline score doesn't translate to online lift.
  • Guardrail metrics matter as much as the target metric — confirm nothing else broke, not just that the number you wanted went up.
  • Re-validate the link periodically — a golden set and its online correlation both go stale as usage patterns shift.

Frequently Asked Questions

What's the difference between offline and online evaluation for LLMs?

Offline evaluation scores model outputs against a fixed, labeled dataset before launch — it's repeatable and answers "is this output correct?" Online evaluation measures real user behavior after launch through A/B tests or holdouts — it's noisier and answers "did this change what users actually did?"

Can a model pass all its offline evals and still fail in production?

Yes, and it happens often. Offline evals check correctness against known references; they can't capture latency-driven abandonment, tone mismatches, novelty effects, or shifts in live traffic that a static golden set never saw. A passing offline score is a necessary filter, not a guarantee of online lift.

Do you need both offline and online evals, or is one enough?

You need both. Offline evals are the only practical way to catch regressions before real users see them; online evals are the only way to confirm the change actually helped. Skipping offline evals means shipping blind; skipping online evals means never learning if what shipped worked.

How often should you re-run online evaluation after launch?

Treat it as continuous, not a one-time check after launch — most teams monitor core online metrics daily or weekly and re-examine the offline-to-online correlation on a longer cadence (monthly or per major model change) since usage patterns and traffic composition drift over time.

What sample size do you need for online AI evaluation metrics to be trustworthy?

It depends on your baseline metric's variance and the effect size you're trying to detect, which is exactly why Kohavi's work on controlled experiments emphasizes power analysis before launching a test rather than eyeballing results — an underpowered test can easily show "no difference" when a real one exists.