Setting SLOs and SLAs for AI Features When the Model Is Non-Deterministic

You can commit to hard numbers on latency, uptime, and error rate for an AI feature, because those are properties of your infrastructure, not the model's judgment. Quality, accuracy, and groundedness cannot be promised the same way — they need probabilistic framing (a target rate on an eval set, with a confidence interval), because the same prompt can produce different outputs on different runs.

Quick Answer: Write an ai feature slo in two tiers — a deterministic tier (p95 latency, uptime, error rate) you commit to like any other service, and a probabilistic tier (quality/groundedness thresholds measured against a versioned eval set) you commit to as a target with a tolerance band, not a guarantee. Treat quality misses like an error budget, not an incident.

Why AI features break the traditional SLA model

Non-determinism means the same request can pass or fail the same quality bar on different calls, which a classic SLA was never built to express. Traditional SLAs describe a system's behavior toward its own promises — a server either responds in 200ms or it doesn't. An LLM's "correctness" isn't a system property; it's a statistical property of a distribution of outputs.

Site Reliability Engineering, as codified in Google's SRE book (Beyoncé Rule: "if you didn't test it, it doesn't work" plus the whole error-budget apparatus), assumes failures are binary and measurable in real time — a 500 status code, a timeout. An AI feature's failure mode is often a plausible-sounding wrong answer, which no load balancer or uptime monitor detects.

This forces a split:

• Infrastructure-layer SLOs — latency, availability, throughput — behave exactly like any other service's, because they're governed by your serving stack, not the model's weights.
• Output-quality SLOs — accuracy, groundedness, tone-adherence — behave like a model evaluation metric, measured periodically against a labeled set, not observed per-request in production the way uptime is.

Conflating the two is the single most common mistake teams make when a sales or legal team asks "what's our SLA for the AI feature." The honest answer is two SLAs, not one.

Which service levels you can honestly promise

Latency, uptime, and error rate are honest commitments because they're measurable per-request, verifiable in real time, and largely under your team's control. These map cleanly onto existing observability tooling — you already have the plumbing (or should).

A useful gut check when a stakeholder pushes for a hard quality number: could you write an automated test that fails a build the instant this promise breaks? If yes (latency, uptime), it's an SLA candidate. If the honest answer is "we'd need a human to review 50 outputs and score them," it's an SLO tracked against an eval set — not a contractual SLA.

Where model routing and caching change the calculus

Latency commitments get easier to keep when you have levers to pull under load. This is where model-routing strategy earns its place in the SLA conversation: a well-designed cheap/fast/default routing strategy for LLM calls lets you degrade gracefully to a faster, cheaper model when p95 latency is at risk, instead of blowing the SLA outright.

Similarly, semantic caching for LLM responses and prompt caching mechanics directly move your latency and cost numbers — a cache hit can turn a 3-second p95 into 200ms. Any latency SLA you write should account for cache hit rate as an assumption, not an afterthought — a cold cache under a traffic spike is exactly when your p95 will spike.

The template SLO: writing it down precisely

A usable AI feature SLO states a deterministic threshold, a probabilistic threshold, and the exact eval mechanism used to measure the second one — vague quality language is not an SLO. Here is a template you can adapt directly:

p95 latency < 3.0s (measured at the API gateway, excluding client network time)
p99 latency < 8.0s
uptime >= 99.5% monthly (excludes scheduled maintenance windows)
error rate < 0.5% (5xx, timeout, malformed-response classes)
groundedness >= 0.90 on eval set `support-answers-v3` (n=500, refreshed quarterly)
factual accuracy >= 0.85 on eval set, human-graded rubric, refreshed quarterly
tone-adherence >= 0.80, LLM-judge scored, calibrated against human raters monthly

Three things make this template honest rather than aspirational-marketing:

1. Named, versioned eval sets. support-answers-v3 is a specific artifact someone can inspect — not "our internal testing." Version it like code, because prompt and model changes will require re-baselining.
2. A refresh cadence. Quality thresholds decay as user behavior drifts (this is data/concept drift, the same phenomenon ML monitoring literature from groups like Google's ML Ops guidance and Andrew Ng's data-centric AI work describes) — a quarterly refresh keeps the eval set representative of real traffic, not a snapshot from launch day.
3. Explicit measurement method. "LLM-judge scored, calibrated against human raters" tells a skeptical engineering lead exactly how the number gets produced and where its own error bars are — an LLM judge is itself non-deterministic, so state the calibration step.

Confidence intervals belong in the number

A single point estimate ("90% groundedness") invites someone to treat 89.4% as a clean SLA breach. Report it as a range: 89-91% at 95% confidence, n=500. This borrows directly from how NIST's AI Risk Management Framework recommends communicating model performance — as a distribution with stated uncertainty, not a single scalar a contract can be litigated over.

Error budgets for AI failure modes

An error budget lets you spend a fixed, pre-agreed amount of quality-shortfall before it triggers a response — exactly like SRE error budgets for downtime, applied to hallucination and drift instead of outages. This reframes "the AI was wrong" from a fire drill into an expected, budgeted event.

The classic SRE error budget: if your SLO is 99.9% uptime, your budget is 0.1% downtime per period — spend it on deploys and experiments, and stop shipping risky changes once it's exhausted. The same structure maps onto AI quality:

The mechanism matters more than the exact numbers: a budget converts every miss into a decision point instead of an argument. When the groundedness eval comes back at 87% against a 90% target, the question isn't "whose fault is this" — it's "have we exceeded this quarter's allotted variance, and if so, what's frozen until we fix it."

Why "zero tolerance" quality SLAs backfire

Teams that promise 100% accuracy, or refuse to put a number on quality at all, tend to end up in one of two bad places: either they quietly under-deliver and erode trust when reality surfaces, or engineering burns effort chasing a threshold that's mathematically unreachable given the model's inherent variance. A stated, monitored error budget is the honest middle path — it's the same tradeoff logic covered in a broader tradeoff-analysis framework for product decisions: naming the tradeoff explicitly beats pretending it doesn't exist.

Negotiating SLAs with stakeholders and customers

Bring the two-tier structure — hard infra SLAs plus probabilistic quality SLOs with error budgets — into every reliability conversation, and frame the quality tier around the user's actual job, not an abstract accuracy score. Grounding the quality bar in what the user is trying to accomplish keeps the negotiation concrete instead of philosophical.

This is where mapping the feature back to a customer jobs-to-be-done analysis earns its keep: a support-answer feature's groundedness threshold should be set by how costly a wrong answer is to the job the customer hired it for, not a generic "90% sounds good" instinct. A billing-dispute answer bot needs a tighter budget than a tone-of-voice rewriter.

It also helps to walk the failure mode across the full customer journey: a hallucination at first-contact onboarding erodes trust differently than the same error rate on an internal, already-skeptical power-user workflow. Set different quality SLOs per journey stage rather than one blanket number across the whole feature.

Practical negotiation script:

1. Open with the deterministic tier — customers accept latency/uptime SLAs readily because they're familiar.
2. Introduce the quality tier as an eval-set-based target with a stated confidence interval, not a guarantee.
3. Name the error budget and what happens when it's exhausted (freeze, escalate, roll back) — this reassures a skeptical stakeholder that a miss has a defined response, not silence.
4. Agree on the refresh cadence for the eval set up front, so "the eval set changed" isn't a surprise renegotiation later.

Keeping the SLA visible through hand-off

Reliability commitments that live only in a Slack thread or a slide deck decay the moment the PM who negotiated them moves to the next project. Keeping the agreed latency and quality thresholds as living requirements attached to the actual spec — visible to engineering at build time, not just at the kickoff meeting — is what keeps a negotiated SLA from quietly eroding after hand-off.

This is the specific gap Prodinja's Spec Studio is built to close: it holds a living PRD with PR-style diffs and readiness gates, so a threshold like "p95 latency < 3s, groundedness > 0.9 on eval set" stays attached to the requirement itself and travels with every subsequent change, rather than getting summarized away into a one-line Jira ticket. Whether that visibility should extend to real-time SLA dashboards is exactly the kind of build-vs-defer call the tradeoff-analysis framework is designed to walk through.

Key Takeaways

• Split every AI feature SLA into two tiers: deterministic (latency, uptime, error rate) and probabilistic (quality, groundedness, accuracy) — never present quality as a hard guarantee.
• Name the eval set, version it, and set a refresh cadence — a quality SLO without a specific, versioned measurement mechanism is just a marketing number.
• Report quality thresholds as a range with a confidence interval, not a single point estimate that invites disputes over a fraction of a percentage point.
• Apply SRE-style error budgets to hallucination and drift, not just downtime — a budgeted miss is a decision trigger, not an incident.
• Use model routing and caching as latency levers so your p95 commitment survives traffic spikes without breaching the SLA.
• Ground quality thresholds in the customer's actual job and journey stage, not a single blanket accuracy number across the whole feature.
• Keep negotiated thresholds attached to the living spec so they survive engineering hand-off instead of decaying into an unenforced slide.