SLOs for AI Quality: Turning 'Good Enough' Into an Operational Promise

An AI quality SLO is a documented commitment that a defined share of your model's outputs will meet a quality bar over a rolling time window — not a one-time launch check. It pairs a measurable SLI (say, groundedness score) with a target (98%) and a window (28 days), then spells out what happens the moment you breach it.

Quick Answer: An AI quality SLO = SLI (what you measure) + target (how good, over what window) + error budget (how much failure you tolerate) + breach policy (what happens when you blow it). A launch threshold is a one-time gate; an SLO is the ongoing promise that keeps that gate meaningful after ship day.

What Is an AI Quality SLO, and Why "Good Enough" Isn't a Strategy

An AI quality SLO (service level objective) is a discipline borrowed from site reliability engineering: a numeric target for how often your AI feature must clear a defined quality bar, measured continuously over a rolling window, with an error budget and pre-agreed consequences for breach. It turns a vague "good enough" judgment call into an auditable commitment.

The vocabulary matters because it's precise where "good enough" is not. In the framing Google popularized through its Site Reliability Engineering discipline (formalized by Ben Treynor Sloss and colleagues), three terms do distinct jobs:

• SLI (indicator): the actual metric you measure — e.g., "% of responses a calibrated judge rates non-hallucinatory."
• SLO (objective): the target value for that metric — e.g., "98% or higher."
• SLA (agreement): an external, often contractual, promise built on top of an SLO — rare for internal AI features, common for AI vendors selling API uptime or accuracy guarantees.

"Good enough" fails as an operating model for four concrete reasons:

1. No enforcement mechanism — nobody's job depends on it staying true.
2. No time horizon — a model that was good enough in January can drift silently by June as underlying providers update models, prompts get patched piecemeal, or retrieval indexes go stale.
3. No owner — when quality slips, there's no named person whose role is to notice.
4. No budget for tradeoffs — teams either ship recklessly or freeze everything, with no principled middle ground.

NIST's AI Risk Management Framework makes a related point directly: it treats AI risk management as continuous, not a pre-deployment checkbox, explicitly calling for ongoing monitoring after a system goes live. A launch-day threshold answers a question that stops being the right question the day after launch. If your evals practice is still forming, the complete guide to AI evals covers the broader discipline an SLO sits inside of.

Launch Threshold vs. Continuous SLO: Two Different Jobs

A launch threshold is a one-time gate — "ship if accuracy is at or above 90% on the golden set" — that answers whether a feature is ready to release. A continuous SLO asks a different question forever: is it still good enough this week, this month, after the last three prompt tweaks? Conflating the two leaves teams confident right up until the moment reality disagrees.

As the guide to numeric thresholds and failable evals lays out, a hard pass/fail bar at launch is necessary but not sufficient — it tells you the model cleared the bar on ship day, not that it's still clearing it three months and several silent model-provider updates later. The threshold was never designed to answer that second question; an SLO is.

The practical takeaway: keep both. A launch threshold is your entry gate; a continuous SLO is the standing contract you sign the moment that gate opens. Skipping the SLO means the only quality check your feature will ever get is the one it passed before anyone was using it for real.

Anatomy of a Quality SLO: SLI, Target, and Window

A well-formed SLO statement has three load-bearing parts — the SLI (a precise, measurable proxy for quality), the target (the percentage or score you commit to), and the window (the rolling period you measure over) — plus a sampling method that makes the resulting number trustworthy. Miss any one part and the SLO is just an opinion with a percentage sign on it.

Here's a sample SLO statement written the way a team would actually publish it:

SLO: 98% of billing-assistant responses score ≥4 out of 5 on the accuracy rubric, evaluated by a calibrated LLM-as-judge against a 300-example stratified golden set refreshed monthly, measured over a trailing 28-day rolling window, with human audit of a 5% random daily sample.

Notice what makes this enforceable rather than aspirational: a named flow (billing-assistant, not "the AI"), a specific rubric score rather than "accuracy" in the abstract, a stated measurement method, and a human audit path that catches judge drift. Building the golden set that anchors this kind of SLI is its own discipline — the walkthrough on building your first golden dataset from 50 examples is a reasonable starting point if you don't have one yet.

Choosing the right SLI is the harder half of this exercise. Common candidates, by feature type:

Deciding whether an LLM-as-judge is trustworthy enough to be your SLI's measurement instrument is a prerequisite, not a footnote — the piece on when to trust an LLM-as-judge walks through the calibration work (agreement rate against human labels, known failure modes) that has to happen before you let a judge's score decide whether you've breached an SLO.

Error Budgets: The Speed-vs-Risk Governor

An error budget is the inverse of your SLO target: if the target is 98%, your budget is the 2% of interactions allowed to fail before you're required to slow down. It's the mechanism that converts an abstract quality bar into an actual shipping-speed lever — burn the budget fast and velocity gets throttled; stay comfortably under it and you've earned room to ship faster or run more experiments.

The math is simple but worth seeing concretely, because "2%" feels abstract until it's translated into how many bad responses that actually allows:

A feature running at high volume burns through a tight budget fast — 100 failures out of 10,000 daily interactions at a 99% target isn't a rounding error, it's the whole month's budget gone in a day if it keeps up. Tracking burn rate (how fast the budget is being consumed relative to the window) rather than just the raw failure count is what turns this into an early-warning system instead of a monthly surprise.

This is also, deliberately, a speed lever rather than a pure quality gate. DORA's long-running research into software delivery performance (Nicole Forsgren and colleagues) treats deployment frequency and change failure rate as two sides of one tradeoff, not competing goals.

Elite-performing teams in that research sustain frequent deploys and low change-failure rates simultaneously, precisely because they've built a governor — not a blanket freeze — into the release process. An error budget is that governor, applied to AI output quality instead of infrastructure uptime.

Practically, this means:

• A team sitting well under budget can ship a riskier prompt change or try a cheaper model.
• A team that's burned most of its budget should be shipping conservative, low-risk changes only — or nothing at all.
• The budget resets on a rolling basis, so a bad week doesn't permanently condemn a feature; it earns back room as the window rolls forward.

What Happens at Breach: Freeze, Rollback, and Escalation Policy

A breach policy is the pre-agreed set of actions triggered the moment your error budget is exhausted — typically a change freeze on non-quality work, a mandatory root-cause review, and standing authority to roll back whatever model or prompt version caused the regression. Without a policy written down in advance, a breach turns into a debate instead of a decision, usually while users are still experiencing the bad output.

A workable breach policy has six components:

1. Detection — a dashboard and alert wired to burn rate, not just the raw SLI, so you're warned before the budget hits zero.
2. Freeze — halt new prompt versions, model upgrades, or feature changes touching the affected flow until the budget recovers.
3. Root cause — a structured error-analysis session on the failing examples, not a guess. Hamel Husain's widely cited essay on AI product evals makes the case that manually reading failure transcripts, one by one, surfaces root causes that aggregate metrics hide — the discipline applies directly here.
4. Remediation — roll back to the last known-good model or prompt version, or patch the specific failure mode (a retrieval fix, a stricter guardrail) if the cause is isolated.
5. Escalation — a named list of who gets notified at breach and who holds authority to override the freeze if there's a genuine business reason to ship anyway.
6. Recovery review — a blameless postmortem once the budget stabilizes, mirroring the postmortem discipline SRE teams run after an availability incident, so the same failure mode doesn't quietly recur next quarter.

The freeze is the part teams most often skip, usually because it's uncomfortable — it means telling a stakeholder "no" for a defined period. That discomfort is exactly what makes writing the policy down before a breach so valuable: the freeze becomes a pre-agreed rule everyone signed up for, not a judgment call one person has to defend in the moment.

Operationalizing SLOs: Ownership, Dashboards, and Where Your Ship Threshold Fits

Operationalizing an SLO means assigning a named owner, wiring the SLI into a dashboard with burn-rate alerts, and scheduling a recurring review — weekly at first, monthly once the feature stabilizes — so the commitment doesn't quietly rot into a forgotten line in a launch doc. The mechanics are less important than making sure someone's job depends on the number staying true.

Two upstream decisions shape which SLI and target actually matter, and both are worth settling before you write the SLO statement:

• Which job is the feature hired to do. A support-triage assistant hired to resolve tickets fast tolerates a different failure mode than one hired to give a confident, correct legal answer — the jobs-to-be-done framing helps separate "wrong but low-stakes" from "wrong and expensive" before you pick a target.
• Where in the journey a breach actually lands. A quality dip during a moment already loaded with anxiety costs more trust than the same dip during a routine, low-emotion step — mapping the customer journey's emotion curve is how you find out which failures are merely annoying versus which ones are relationship-ending.

If you already set a numeric bar to decide whether a feature is ready to ship, you've done half the SLO work already. Prodinja's prototype experience is built around walking a PM through exactly that judgment call — an Evals critique that helps land on a defensible ship-gate number grounded in a golden dataset rather than a gut call.

The natural next step, once that number exists, is to stop treating it as a one-time gate. Start treating it as your first SLO target instead: same metric, same threshold, now measured continuously instead of once at ship day.

Key Takeaways

• An AI quality SLO turns "good enough" into a number with an owner, a window, and a consequence — SLI (what's measured) + target (the bar) + window (the period) + breach policy (what happens next).
• Launch thresholds and continuous SLOs answer different questions — one gates a release, the other keeps watching after the gate opens; skipping the second leaves quality unmonitored the moment a feature ships.
• A trustworthy SLO needs a real measurement instrument, whether that's a calibrated LLM-as-judge, a refreshed golden dataset, or a human audit sample — an ungrounded SLI produces an ungrounded number.
• Error budgets convert quality into a shipping-speed lever, not just a compliance metric — a comfortable budget earns room to move faster; a nearly-exhausted one should slow the team down before users pay for it.
• Breach policy has to be written down before the breach happens — freeze, root-cause session, rollback authority, and escalation contacts agreed in advance, so a bad week becomes a decision, not a debate.
• The SLI you pick should trace back to the job the feature does and where it sits in the customer journey — the same failure rate means very different things depending on the stakes of the moment it occurs in.