When your AI system produces harmful, biased, or confidently wrong output at scale, the first hour determines whether this becomes a contained incident or a trust-destroying story. Run it like an SRE outage with three AI-specific twists: severity is scored by harm and reach, not uptime; the fix is often a model or prompt version, not a code deploy; and the postmortem has to answer who the output could have hurt.
Quick Answer: Score AI incidents on a harm-and-reach rubric, not downtime. Decide kill-switch vs. rollback vs. throttle inside the first 15 minutes using a pre-agreed decision tree. Assign comms and legal roles before the incident, not during it. Close with a blameless postmortem that names the failure mode, not the person.
How AI Incidents Differ From a Traditional Outage
An AI incident differs from a classic outage in three ways: severity tracks harm and reach instead of uptime, the remediation is usually a model, prompt, or retrieval-source version rather than a code rollback, and a failure can be statistically rare in aggregate yet catastrophic for the one person it hit. A 99.9% "accuracy" rate still means one in a thousand users got something false, offensive, or dangerous.
Classic site-reliability engineering, as codified in Google's Site Reliability Engineering book, built its incident discipline around error budgets and mean time to recovery (MTTR). That model still applies to latency and downtime. It says almost nothing about a chatbot that fabricates a refund policy, a resume screener that silently deprioritizes one demographic, or a coding agent that pushes a destructive command. Those are content and behavior failures, and they need their own taxonomy.
Frameworks worth borrowing from directly:
NIST AI RMF(the NIST AI Risk Management Framework) structures AI risk work into Govern, Map, Measure, and Manage functions — Manage is effectively your incident response phase.MITRE ATLAScatalogs adversarial techniques against AI systems (prompt injection, model extraction, data poisoning), useful for classifying whether an incident is an organic failure or an attack.- The AI Incident Database, maintained by the nonprofit Responsible AI Collaborative, has logged thousands of publicly reported AI harm events since 2014, with the pace of new reports accelerating sharply since generative AI's mainstream adoption in 2023 — evidence that "it won't happen to us" is not a plan.
If your organization already has a responsible-AI operating model, this playbook slots into the incident-response layer of it; see our complete guide to responsible AI for product teams for how governance, evaluation, and incident response fit together as one system rather than three disconnected checklists.
A Severity Rubric for AI Failures
Score AI incidents on two axes — harm severity and reach — the same way PagerDuty-style frameworks score business impact, but replace "revenue at risk" with "who was hurt and how badly." A SEV1 output shown to ten users can outrank a SEV3 shown to ten thousand.
| Severity | Definition | Example trigger | Response SLA | Decision authority |
|---|---|---|---|---|
SEV1 — Critical | Physical, financial, legal, or safety harm; discriminatory outcome affecting a protected class at scale | Medical or financial advice that could cause real-world harm; biased hiring/lending decisions surfaced in production | Page on-call in minutes; kill switch considered immediately | Incident Commander + Legal/Trust & Safety, real time |
SEV2 — High | Harmful, hateful, or clearly false content reaching real users; reputational risk | Chatbot outputs a slur, fabricated policy, or invented citation shown to customers | Response within 30–60 minutes | Incident Commander, with Comms notified |
SEV3 — Moderate | Degraded quality or narrow bias visible to a subset of users, no active harm yet | Model consistently underperforms for one language or user segment | Response within a business day | On-call PM / eng lead |
SEV4 — Low | Internal-only or edge-case failure, no user-facing exposure | Eval regression caught in staging, red-team finding pre-launch | Tracked in backlog | Team lead |
Two calibration notes matter more than the table itself. First, bias and fairness failures are systematically under-scored because they rarely produce a single dramatic ticket — they surface as a slow drift in aggregate metrics. If your team runs a structured AI fairness audit, route its findings into this same rubric instead of a separate "quality" backlog, or fairness regressions will always lose the prioritization fight against a louder SEV2.
Second, reach multiplies severity, and reach is often unknown at detection time. Treat an ambiguous incident as one severity level higher than the evidence in hand until you've confirmed scope — it's far cheaper to stand down a false SEV1 than to under-react to a real one.
The Kill-Switch Decision: Rollback, Throttle, or Watch
Inside the first 15 minutes, choose one of four levers — full kill switch, rollback to a prior model or prompt version, throttle via a feature flag to a smaller cohort, or watch-and-log — using ongoing-harm and reproducibility as the deciding factors, not how disruptive the fix will feel.
Ask these questions in order:
- Is the harm ongoing right now, or was it a one-time output? Ongoing harm forces a kill switch or throttle; a one-time output that isn't reproducible can often be watched while you investigate.
- Can you reproduce it? A reproducible failure mode is safe to root-cause quickly; a non-reproducible one is scarier precisely because you can't bound it — lean toward throttling exposure until you can.
- What's the blast radius? One tenant, one locale, one user cohort, or everyone? Feature flags that segment by cohort let you cut exposure without a full outage.
- Is there a known-good fallback? A previous model version, a rules-based fallback, or a "human review required" gate all count. No fallback means the kill switch is the only responsible option.
| Lever | Speed | Blast-radius reduction | User-visible impact | Best for |
|---|---|---|---|---|
| Full kill switch | Fastest (minutes) | Total | Feature disappears entirely | SEV1, harm is ongoing and severe |
| Rollback to prior model/prompt version | Fast (minutes–hours) | Total, but preserves the feature | Feature keeps working, quietly older | Regression traced to a specific deploy |
| Feature-flag throttle to a cohort | Fast (minutes) | Partial, by segment | Some users unaffected, others revert | Harm isolated to a known segment |
| Human-in-the-loop review gate | Moderate (hours) | Partial, adds friction | Slower responses, added trust signal | High-stakes outputs, ongoing investigation |
| Watch-and-log | N/A — no change | None | None | Confirmed one-off, not reproducible, low severity |
Write the flag names down before you need them — model_v4_enabled: false decided in a panic at 2 a.m. is where typos happen. A pre-agreed decision tree, reviewed quarterly, turns this from a debate into a lookup.
Comms Responsibilities in the First Hour
Assign four standing roles before an incident ever happens — Incident Commander, Comms Lead, Trust & Safety/Legal Liaison, and Technical Lead — adapted from the incident-command structure Google and other SRE-mature organizations use, with the legal liaison made mandatory rather than optional for anything involving AI-generated harm.
- Incident Commander owns the timeline, the severity call, and the kill-switch decision. One person, not a committee.
- Comms Lead drafts internal updates (a running Slack/incident channel), external user-facing messaging (in-app banner, status page, direct email to affected users), and an executive brief — three different documents, three different levels of detail.
- Trust & Safety/Legal Liaison determines whether the incident triggers a disclosure obligation. If your product operates as a high-risk AI system under the EU AI Act, certain serious incidents carry formal reporting timelines to regulators — our EU AI Act map for product managers breaks down which categories of AI system fall under that obligation and what "serious incident" means in the regulation's own terms.
- Technical Lead runs root-cause investigation in parallel with the Commander's containment decision — containment and diagnosis should never block on each other.
Internal comms should over-communicate; external comms should under-promise. Tell your own team every update, even "no change." Tell affected users only what you can stand behind an hour from now.
The most common comms failure isn't silence — it's a premature root-cause claim that gets walked back publicly. Say what happened and what you're doing about exposure now; save causal explanation for the postmortem, where you'll actually have it right.
Walkthrough: A Hallucination-at-Scale Incident From Detection to Remediation
Here's a realistic scenario end to end, the shape most product and trust & safety teams will eventually face: an internal expense-report assistant starts confidently citing the wrong reimbursement policy limits after a routine retrieval-source update, and no one notices for a weekend.
T+0 (Saturday, 9 p.m.): A retrieval-augmented policy assistant, built to answer "what's my meal reimbursement cap on a client trip," starts citing a draft policy document that was accidentally left in the indexed source folder instead of the published one. The number it quotes is roughly double the real cap.
T+14 hours (Sunday, 11 a.m.): Support tickets tagged "policy question" spike three times above baseline. An on-call PM notices the anomaly in the ticket-volume dashboard, not in a model-quality dashboard — a reminder that detection often comes from downstream signals, not the AI system itself.
T+14h 20m: The PM checks the severity rubric. Financial exposure, real users, reproducible on every query about that specific limit — this scores SEV2, trending toward SEV1 if the inflated number leads to approved overpayments. Incident Commander is paged.
T+14h 35m: Kill-switch decision, using the four-question checklist: harm is ongoing (yes), reproducible (yes, deterministically), blast radius is one policy topic across all users (moderate-wide), fallback exists (the previous, correct document). Decision: rollback the retrieval source to the last known-good index, not a full kill switch — the assistant is useful for everything else.
T+15h: Comms Lead posts an in-app banner on the specific policy topic ("This answer may be inaccurate — confirm with Finance before submitting") and emails users who received the inflated figure in the last 48 hours, since some approvals may already be in flight.
T+2 days: Root cause confirmed. A document-sync job promoted a draft file because its filename matched the production naming pattern, and there was no confidence signal or source-freshness check on the retrieval layer — so the assistant stated the wrong number with the same tone of certainty it uses for correct ones.
This is exactly the gap our piece on designing honest confidence and uncertainty into AI UX addresses. A visible "last verified" date or confidence indicator on policy-sensitive answers would have given users a reason to double-check before acting.
T+5 days: Blameless postmortem published, action items assigned: source-freshness validation on the indexing pipeline, a confidence threshold below which the assistant defers to a human, and a monitoring rule that treats "ticket-volume spike on a topic" as a first-class AI-incident signal, not just a support-ops metric.
The Blameless Postmortem Template
A blameless postmortem documents the timeline, contributing factors, and systemic fixes without assigning individual fault — a discipline borrowed directly from Google's SRE postmortem culture — with two AI-specific additions: a harm assessment and an affected-user remediation log.
Use this structure for every SEV1–SEV3 incident:
- Summary — one paragraph, written for someone with no context.
- Impact — who was affected, how many, and how (not just "500 users," but what happened to them and what job they were trying to get done when it went wrong — a Jobs to Be Done lens is useful here for stating impact in terms of the outcome the user actually needed, not just the feature that broke).
- Timeline — timestamped, factual, no interpretation mixed in.
- Detection — how you found out, and how long that took (your
MTTD, mean time to detection). - Root cause — a 5 Whys pass or equivalent, stopping at the systemic cause, not the last person who touched the code.
- Contributing factors — the conditions that made the failure possible (missing eval coverage, no confidence gate, stale documentation).
- What went well / what went wrong / where we got lucky — three honest columns, including luck, which most postmortems omit and shouldn't.
- Harm assessment — severity per the rubric above, plus whether any affected user needs direct remediation (a correction, a refund, an apology, a regulatory notification).
- Action items — each with a named owner and a date, tracked to closure like any other engineering commitment.
Mapping the impact section against a customer journey view — where in the flow the bad output surfaced, and what the user did next — often reveals whether the real fix belongs in the model, the UI, or an earlier step that should have caught the error before it reached the user at all.
Blameless doesn't mean consequence-free. It means the investigation optimizes for finding the systemic fix, not for finding someone to blame — because a team that fears blame stops reporting near-misses, and near-misses are your cheapest source of signal.
Where to capture the calls made under pressure
The hardest part of writing an honest postmortem is reconstructing what people actually knew and decided in the moment — memory reconstructs incidents as more orderly than they were. This is one place a lightweight, real-time capture habit pays off.
Prodinja's Journals module supports real voice capture, so a PM mid-incident can record a fast Friction entry — "policy assistant citing wrong cap, rolling back retrieval source, paging legal" — without stopping to type. Its Decision Journal view then preserves the calls made under pressure in the order they actually happened, which is exactly the raw material a blameless postmortem needs later.
Key Takeaways
- Score AI incidents on harm and reach, not uptime — a rare failure hitting the wrong user can outrank a common one that's merely annoying.
- Decide kill-switch vs. rollback vs. throttle within the first 15 minutes using a pre-agreed checklist: ongoing harm, reproducibility, blast radius, and fallback availability.
- Assign Incident Commander, Comms Lead, Legal/Trust & Safety Liaison, and Technical Lead before an incident happens, not during one — role confusion costs the first 20 minutes.
- Bias and fairness failures are chronically under-scored because they surface as slow drift, not a single dramatic ticket — route fairness audit findings into the same severity rubric as everything else.
- External comms should under-promise; internal comms should over-communicate — resist stating a root cause publicly before the postmortem confirms it.
- Blameless postmortems need a harm assessment and an affected-user remediation log on top of the standard SRE template — AI incidents have victims a code outage usually doesn't.
- Capture decisions as they happen, not after — reconstructed timelines smooth over the uncertainty that was actually driving the calls made under pressure.
Frequently Asked Questions
What is an AI incident, exactly?
An AI incident is any case where a model's output or an AI-driven action causes real or credible harm, embarrassment, or material error for a user, distinct from a system-availability outage. That includes harmful or biased content, confidently stated falsehoods, privacy leakage, and an autonomous agent taking a wrong or destructive action.
How is AI incident severity different from IT incident severity?
Traditional IT severity is scored on downtime and business impact; AI incident severity is scored primarily on harm and reach to the people affected. A model that's fully "up" but confidently wrong to a protected class of users is a higher-severity event than a feature that's briefly unavailable to everyone.
When should we pull the kill switch versus just rolling back?
Pull the full kill switch when harm is ongoing, severe, and you have no safe fallback; roll back to a prior model or prompt version when the feature itself is sound but a recent change introduced the failure. Throttling via feature flags is the middle option when the harm is isolated to a known user segment.
Do AI incidents carry legal or regulatory reporting obligations?
Sometimes, and it depends on jurisdiction and risk classification — under the EU AI Act, providers and deployers of certain high-risk AI systems face formal serious-incident reporting timelines to regulators. Loop in your Trust & Safety/Legal liaison early enough to make that call within the first hour, not after external comms have already gone out.
How long should a blameless postmortem take to publish?
Aim for a draft within 3-5 business days of containment for a SEV1 or SEV2 incident, while the timeline is still fresh but after the immediate firefighting has settled enough to think clearly. Waiting longer lets memory smooth over the genuinely uncertain moments that make the postmortem useful.