Alerting on a nondeterministic system means you never page on a single bad output — you page when the rate of bad outputs moves outside its normal band. A single hallucination is expected noise; a hallucination rate that triples in an hour is a signal. Good alert thresholds for AI systems are set on distributions and windows, not on individual events.

Quick Answer: Don't alert on "the model gave a wrong answer." Alert on "refusal rate is up 3x over a rolling 1-hour window versus its 7-day baseline." Threshold the rate of failure, not the occurrence of failure.

An LLM system is designed to be probabilistic — the same prompt can produce different outputs on different runs, and some fraction of those outputs will be wrong, hedged, refused, or off-tone even when nothing has broken. That's the baseline condition, not an incident. Traditional monitoring assumes a system in steady state that occasionally throws an error; AI systems are in a permanent low-grade state of "occasionally throwing an error" by design. If you page on that baseline, you'll page constantly, and the team will do what every team does with a noisy pager: mute it.

This is the core mental shift a platform PM has to make when writing alerting policy: you're not watching for a broken thing, you're watching for a change in the shape of a distribution that's always had some failure in it. That distinction determines almost everything downstream — what you measure, how you window it, and what actually deserves a 2am page.

Why threshold-on-a-value fails for AI systems

A threshold-on-a-value alert ("page if latency > 2s" or "page if this response contains a refusal phrase") assumes each event is independently meaningful. For nondeterministic systems, most single events are not — they're draws from a distribution that always includes some bad outcomes, so alerting on one draw is statistically equivalent to alerting on noise.

Classic infrastructure monitoring works on value thresholds because the underlying system is close to deterministic: a healthy database doesn't return a 500 error under normal load, so any 500 is informative. An LLM, by contrast, routinely produces a bad completion under normal load — hallucination rates in retrieval-augmented systems commonly sit in the low single digits to low teens percent even when everything is functioning as designed, per findings summarized in Stanford's annual AI Index Report. Zero-tolerance thresholds don't match that reality.

Three failure modes follow directly from applying value thresholds to a probabilistic system:

  1. Constant false pages. Every model in production produces some bad completions; alerting on any single one means the pager fires continuously, and continuously-firing pagers get muted, silenced, or routed to a channel nobody reads.
  2. Missed real incidents. A genuine degradation — a prompt-injection vulnerability, a context-window regression, a bad model version rollout — usually shows up as a shift in the rate of a metric, not one dramatic single failure. A value-threshold alert tuned to ignore normal noise will also ignore the early signal of a real shift, because it isn't looking at rate at all.
  3. No sense of "how much worse." A single-event alert can't tell you if today is marginally worse than yesterday or catastrophically worse — it just tells you "this one thing happened," which strips out the magnitude information a responder actually needs to triage.

The fix isn't a smarter single-event filter. It's changing what you're comparing: not "did this event violate a fixed line," but "does this window's rate deviate from its recent normal." That reframing is what the rest of this piece works through — and it's the same underlying idea covered in more general terms in a complete guide to LLMOps observability, which frames rate-based alerting as one leg of a broader monitoring stack.

Building the distribution: windows, baselines, and rates

A distribution-based alert has three moving parts: a metric (what you count), a window (how you group counts into a rate), and a baseline (what "normal" looks like for that rate). Get any one wrong and the alert either fires on noise or sleeps through drift.

Pick metrics that are rates, not events

Start from metrics that are naturally expressed as a proportion of traffic, not a single observation. Useful candidates for most LLM products:

MetricWhat it capturesTypical healthy range*
Refusal rateShare of requests the model declines to answer1-5%
Hallucination / groundedness failure rateShare of responses not supported by provided context2-12%
Escalation-to-human rateShare of sessions routed to a person3-10%
Negative feedback rateShare of responses thumbs-downed or flagged1-6%
Retry/regeneration rateShare of turns where the user re-prompts immediately5-15%

*Ranges are illustrative starting points, not universal targets — your own baseline, measured over your own traffic, is the only range that matters. This is exactly the point: you calibrate the "normal" band from your own history, the same way the quality drift without a stack trace framing argues that AI failure has no fixed signature to match against.

Windowing: the size of your rolling bucket

The window is the denominator that turns raw counts into a rate you can trust. Too short a window and low-traffic periods produce statistically meaningless spikes — five refusals out of eight requests at 3am looks like a 60% refusal rate but is really just small-sample noise. Too long a window and you smear a real, fast-moving incident into invisibility.

  • Short windows (5-15 minutes) suit high-volume endpoints where you have enough requests per minute for a rate to be statistically stable; good for catching a bad deploy fast.
  • Medium windows (1 hour) are the workhorse for most product-level alerts — refusal rate, hallucination rate, escalation rate — because they smooth out bursty traffic while still catching same-shift degradations.
  • Long windows (24 hours, 7 days) are for baselines, not alerts — you compare a short window's rate against a long window's trailing average, not against another short window.
  • Always require a minimum sample size before a window's rate counts at all (e.g., "don't alert on this window unless it saw at least 200 requests") — this is the single most common gap in early AI alerting setups and the direct fix for the 3am small-sample problem above.

Moving baselines beat fixed thresholds

A fixed threshold ("alert if hallucination rate > 8%") is really just a value threshold wearing a rate's clothing — it will drift out of date the moment your traffic mix, model version, or user base shifts, and nobody remembers to revisit it. A moving baseline compares the current window's rate to a trailing average (commonly 7 or 14 days) of the same metric, then alerts on the ratio or standard-deviation distance between them.

Two common baseline approaches, each with a real tradeoff:

  • Trailing rolling average with a multiplier ("alert if this hour's rate is 3x the trailing 7-day average for the same hour-of-week") — simple to implement and explain, but slow to adapt after a deliberate, expected shift (a new product launch that changes normal behavior).
  • Statistical control limits (mean ± k standard deviations, the same logic behind statistical process control popularized by Walter Shewhart's control charts) — more sensitive to genuine drift and self-adjusting as variance changes, but requires enough historical data to estimate variance reliably and is harder to explain to a non-technical stakeholder in a postmortem.

Either way, the baseline needs periodic recalibration — a model upgrade, a prompt change, or a new user segment can shift what "normal" looks like, and a stale baseline either goes silent or screams constantly. This is a policy-owner job, not a one-time engineering task, which is exactly where a recurring review cadence earns its keep (more on that below).

A good alert vs. a bad alert, side by side

The clearest way to internalize distribution-based alerting is to compare a bad, event-triggered alert against a good, rate-and-window alert built on the same underlying failure mode. The difference isn't sophistication — it's what gets compared to what.

DimensionBad alertGood alert
Trigger"A user received a hallucinated answer""Hallucination rate over the last 1h is 3x the trailing 7-day baseline"
Unit of comparisonSingle eventRate vs. moving baseline
Sample size1Requires minimum N requests in window
Expected fire frequencyEvery shift, all the timeRare, only on genuine deviation
Responder's next stepUnclear — investigate one output?Clear — check what changed (model version, prompt, traffic mix) around the window
Fatigue riskHigh — mutes within a weekLow — each page has evidence behind it

The bad alert isn't wrong because hallucinations don't matter — it's wrong because it treats a known, expected failure mode as if any instance of it were exceptional. The good alert encodes the actual claim you care about: something changed, not something happened. That's the same distinction the three dashboards every AI product needs on day one draws between a dashboard that shows raw events and one that shows rates against baseline — alerting is just the automated, paging version of the same discipline.

Multi-condition alerts reduce false positives further

Stack a second, unrelated signal onto a rate spike before paging, and you cut false positives without dulling sensitivity to real incidents. For example: "hallucination rate is 3x baseline and negative feedback rate is also elevated and the spike isn't isolated to a single low-traffic segment." Requiring corroboration across independent metrics is a standard technique in anomaly detection more broadly, and it directly answers the objection that a single metric's rate can spike for a boring reason (a traffic-mix shift, a new user cohort with different intent) rather than a real quality regression.

Designing for alert fatigue from day one

Alert fatigue is the default failure mode of any alerting system for a probabilistic product, and it compounds: the more “safe” alerts you ship, the faster real ones get ignored. Google's Site Reliability Engineering practice, formalized in the SRE book from Google, treats this as central — every alert should map to an action a human can actually take, or it shouldn't page at all.

Concretely, that means applying a short discipline to every candidate alert before it ships:

  1. Does this alert map to a specific action? If the runbook for a page is "look at it and decide," it's not ready to page — it's a dashboard panel, not an alert.
  2. Is severity tiered? Not every deviation deserves a 2am page. A 1.5x rate deviation might warrant a Slack notification during business hours; a 5x deviation with corroborating signals warrants an on-call page.
  3. Does the alert self-resolve or auto-suppress during known events? A planned model upgrade, an A/B test, or a traffic surge from a marketing push will all shift rates temporarily — suppress or annotate alerts during known changes so responders aren't debugging an expected shift.
  4. Is there a maximum page rate per on-call shift? If a policy is generating more than a small handful of pages per week, the thresholds are miscalibrated, not the system — that's a signal to recalibrate the baseline, not to keep paging.
  5. Who owns recalibrating this threshold, and how often? Thresholds rot. A baseline set six months ago on last year's traffic mix and last year's model version is quietly wrong today, and nobody notices until it's either silent through a real incident or screaming through a benign shift.

That last point is where alerting policy stops being a one-time configuration task and becomes a recurring operational habit — which is also where most teams quietly drop the ball, because nothing forces the review to happen.

Where Prodinja fits into the recalibration habit

Prodinja doesn't run your alerts or compute your thresholds — that logic lives in your observability stack. What it's designed to hold is the cadence: a Prodinja Reminder set to recur monthly or quarterly, tied to a specific stakeholder or on-call owner, that surfaces "recalibrate the hallucination-rate baseline" or "review pages fired this quarter for signal vs. noise" before the review quietly stops happening. It's a lightweight way to make sure a threshold that made sense in month one doesn't silently rot into either constant noise or dead silence by month six.

From alert to fix: closing the loop

An alert that fires and gets dismissed is worse than no alert — it trains responders to distrust the pager and teaches nothing to the underlying system. A distribution-based alert should terminate in one of two places: a confirmed transient blip (annotate and move on) or a durable case added to your eval suite so the same regression is caught automatically next time, before it needs a human page at all.

That second path — an alert firing today becoming a reusable regression check tomorrow — is the same discipline the guide to closing the loop from failures to eval cases covers in depth: every page that turns out to be a real regression should leave behind an eval case, not just a resolved ticket. Otherwise you've fixed the symptom this week and left the system exposed to page on the identical cause next quarter.

Key Takeaways

  • Alert on rate, not on events — a single hallucination or refusal is expected background noise in a probabilistic system; a shift in the rate of either is the actual signal.
  • Window your metrics with a minimum sample size so a handful of failures during a quiet traffic period doesn't masquerade as a spike.
  • Compare against a moving baseline, not a fixed threshold, because "normal" shifts every time you change a model version, a prompt, or your user mix.
  • A good alert names a magnitude and a comparison ("3x over 1h vs. 7-day baseline"), while a bad alert names a single occurrence.
  • Tier severity and cap page volume per shift — if a policy generates more than a handful of pages a week, the thresholds need recalibrating, not the system.
  • Every real page should end in an eval case, not just a resolved ticket, so the same regression gets caught automatically next time.
  • Thresholds need a recurring owner and cadence, since a baseline set once against last quarter's traffic will quietly drift wrong — this is a policy-maintenance job, not a one-time config task.

Frequently Asked Questions

What's the difference between alerting on AI systems versus traditional software?

Traditional software alerting assumes a mostly-deterministic system where any error is informative on its own. AI systems are designed to produce some share of imperfect outputs even when healthy, so the useful signal is a change in that share over time — a rate shift, not a single event.

How do you set alert thresholds for AI systems without historical data?

Start conservatively wide (e.g., 3-5x a rough baseline) using whatever data you have, even a few weeks, then tighten as you accumulate a real trailing history. It's better to under-alert for the first month than to ship a threshold that pages on ordinary variance and gets muted before it's calibrated.

How often should alert thresholds be recalibrated?

Most teams find monthly-to-quarterly recalibration sufficient, with an unscheduled review after any model version change, major prompt update, or traffic-mix shift. A recurring reminder tied to a named owner is the practical mechanism that keeps this from lapsing.

What causes alert fatigue in AI monitoring specifically?

Alert fatigue in AI monitoring almost always traces back to value-threshold alerts on inherently noisy metrics — paging on a single hallucination, refusal, or low-confidence score that would have happened anyway under normal operation, so the page carries no new information.

Should every anomaly in an AI system trigger a page?

No — only anomalies with a clear action attached should page; lower-confidence or lower-severity deviations belong in a dashboard or a business-hours notification, reserving pages for deviations large enough, and corroborated enough, that someone needs to act immediately.