When a product breaks in production, in front of paying customers, how you lead the next ninety minutes matters as much as the fix itself. Name one incident commander with real authority to decide, staff a handful of clear roles beneath them, and choose fast containment over five stakeholders half-deciding in a shared chat.

Quick answer: Appoint a single incident commander who owns every call during the crisis, assign clear supporting roles instead of an open group chat, communicate on a fixed cadence even with no news, and review decisions afterward against what was knowable at the time — not what hindsight now makes obvious.

Why Most Product Crises Get Worse Before They Get Better

Crises rarely escalate because the underlying problem was unsolvable. They escalate because nobody had unambiguous authority to decide, so five well-meaning people half-owned the call and the group defaulted to more meetings instead of action. Every additional decision-maker in the room multiplies coordination cost faster than it multiplies insight.

The instinct at a director or VP level is to gather everyone who might have context — engineering leads, support, legal, comms, a founder or two — into one call. That instinct is right for information-gathering and wrong for decision-making. A crisis needs a funnel, not a roundtable.

This is precisely the problem the Incident Command System (ICS) was built to solve. Developed by California fire agencies in the 1970s after a string of wildfires overwhelmed departments trying to coordinate through ad hoc, multi-agency committees, ICS gives every incident — a wildfire, a hurricane, a chemical spill — exactly one commander and a fixed set of roles beneath them. The U.S. Federal Emergency Management Agency later adopted it nationally because uncoordinated command by committee kept failing the same way regardless of the disaster type.

Product crises are smaller in scale but structurally identical. A payment outage, a data exposure, a viral complaint about a broken feature — each needs the same shape of response: one owner, defined roles, and a bias toward decisive containment over consensus. Borrowing that shape doesn't require a wildland-fire manual. It requires three commitments made before your next fire, not during it.

Borrowing Incident Command: Roles Before the Roles Blur

A crisis without assigned roles doesn't stay roleless — it silently assigns them anyway, usually to whoever talks loudest or panics quietest. Decide the roles in advance, in a calm room, so nobody is negotiating org-chart questions while a system is down and customers are posting screenshots.

At minimum, five roles cover most product crises:

RoleOwnsTypically staffed by
Incident CommanderThe final call on every containment decision; can override any other roleDirector/VP or senior PM with cross-functional standing
Operations LeadTechnical diagnosis and the actual fixSenior engineer or eng manager closest to the system
Communications LeadAll external and internal messaging, on a fixed cadencePM, comms, or support lead — never the IC
ScribeA timestamped log of every decision and the information behind itAny team member not needed elsewhere
Customer LiaisonDirect line to the highest-impact affected accountsCSM, sales, or account lead

The incident commander is not automatically the most senior person on the call, and not automatically the person who understands the bug best. It's whoever can hold the full picture and make an irreversible-feeling call under incomplete information without deferring it back to the group. Seniority helps only insofar as it buys authority to be believed.

Assigning a scribe feels bureaucratic mid-crisis and pays for itself entirely at the post-incident review, covered later. Without a live decision log, the retro degenerates into competing memories reconstructed under hindsight bias.

Leadership style matters here too. Daniel Goleman's research on six leadership styles — laid out in our complete guide to leadership styles in the room — found that a commanding style, used briefly, is the right tool during acute crises, even for leaders who default to coaching or democratic styles the rest of the year.

The mistake isn't being directive during the fire. It's staying directive after it's out. For the fuller leadership context this playbook sits inside, see our complete guide to PM leadership.

The First Sixty Minutes: Decisive Containment Over Consensus

In the opening hour, the job is containment, not consensus, and not root-cause certainty. Stop the bleeding with the best available information, log the confidence level behind that call, and resist the pull to wait for a fuller diagnosis before acting.

A useful discipline: separate a containment decision (stop the damage now) from a fix decision (resolve the underlying cause), and never let the second block the first. Rolling back a bad deploy is a containment decision available in minutes; understanding why it broke is a fix decision that can take hours. Do both, in that order.

  1. Declare the incident and name the commander — out loud, in writing, within minutes of confirming something is actually wrong, not merely reported wrong.
  2. Staff the four supporting roles from whoever is available; don't wait for the "right" person if a capable one is present.
  3. Make the containment call on the best information in the room, and have the scribe log it with a stated confidence level.
  4. Communicate immediately — even a message that says only "we know, we're on it, next update in 30 minutes" beats silence.
  5. Set a fixed check-in cadence, commonly every 30-60 minutes, rather than updating whenever something changes.

The consensus-run alternative fails for a specific, structural reason: it optimizes for nobody individually being wrong, which is a different goal from the system getting fixed fast.

DimensionCommittee-run responseIncident-command response
Decision speedSlow — waits for agreementFast — one owner decides
AccountabilityDiffuse; hard to trace a call to a personClear; every decision has a named owner
CommunicationReactive, inconsistent cadenceScheduled, predictable cadence
Morale under pressureErodes — everyone feels responsible for everythingHolds — each role has a bounded job
Post-incident learningReconstructed from memory, contestedLogged in real time by the scribe

As one long-standing operations maxim puts it: in a crisis, a decent decision made now beats a perfect decision made later — because "later" is where the damage compounds.

Communicating Through the Fire: Calm, Frequent, Honest

The single biggest reputational risk in a crisis usually isn't the outage itself — it's silence, or worse, a confident update that turns out to be wrong. Communicate early, on a fixed schedule, and say plainly what you know, what you don't, and when you'll update again.

Three disciplines make this work under pressure:

  • Cadence over completeness. A short "still working on it" every 30 minutes builds more trust than a long, polished update every three hours. Silence reads as either incompetence or dishonesty, even when the team is working hard.
  • Calibrated language. Say "we believe" and "we're investigating" when that's true; reserve "confirmed" and "resolved" for when they're actually true. Overcorrecting a status update erodes trust faster than the original incident did.
  • Separate audiences, same facts. Customers, internal stakeholders, and the board need different detail levels but never different underlying claims — a comms lead reconciling three versions of "what happened" mid-crisis is a self-inflicted second incident.

It helps to think about the affected customer's experience as its own emotional arc rather than a single ticket. The same customer journey emotion curve used in normal product work — mapped in our complete guide to customer journey mapping — applies under crisis conditions too: frustration peaks fastest right after the first symptom, and a fast, honest first response blunts that peak more than a technically superior fix delivered silently an hour later.

It's also worth asking, mid-crisis, what job the customer was actually trying to get done when your product failed them — the same lens behind Jobs to Be Done theory. A checkout failure during a customer's one attempt to buy a gift is a different severity than the same failure during routine, low-stakes browsing, even at an identical error rate.

The Post-Incident Review: Judge the Decision, Not the Outcome

A good post-incident review asks whether each call was reasonable given what was knowable at the time — not whether it happened to work out. Conflating the two is the single most common way organizations punish good judgment and reward lucky guesses, then wonder why decision quality never improves.

This distinction has a name in decision science: resulting, the error of judging a decision purely by its outcome rather than its process, a pattern closely tied to Daniel Kahneman's research on hindsight bias. Once an outcome is known, people systematically overestimate how predictable it was beforehand — which makes an honest retro nearly impossible without a contemporaneous record of what was actually known when the call was made.

Google's site reliability engineering practice, and the broader blameless postmortem movement it helped popularize, built its process around exactly this distinction: the review exists to fix systems and calibrate judgment, not to identify someone to blame. A blameless review still names decisions and their owners; it just evaluates them against the information available in the moment, not against the fully-resolved timeline everyone can see afterward.

Three questions structure a review that actually improves the next crisis:

  1. What did the incident commander know, and when, at each containment decision?
  2. Was the confidence level stated at the time consistent with how it played out — a low-confidence call that happened to work is still a low-confidence call.
  3. What would a reasonable commander with the same information have done differently, if anything at all?

This is exactly where a contemporaneous decision log earns its keep. Prodinja's Decision Journal lets you log the fast, high-stakes calls made mid-crisis, along with the confidence level attached to each one at the moment it was made — so the retro can separate a good process from a merely lucky outcome instead of relying on reconstructed memory. Our guide to calibrating judgment with a decision journal goes deeper on the habit outside of crisis conditions too.

Occasionally a review surfaces something other than a process gap — a role holder who genuinely wasn't equipped for the call, repeatedly, across incidents. That's a different, harder conversation, and it deserves the same care as the crisis itself; our guide on managing someone out with dignity covers how to handle it without turning a system failure into a scapegoat hunt.

When the Fire Is Your AI: A Different Playbook

Not every product crisis is an incident-command crisis in the classic sense, and an AI system confidently generating wrong output is the clearest exception. A hallucinating model rarely announces itself the way an outage does — there's no dashboard alert, no 500 error, just plausible-sounding wrongness sitting inside otherwise-normal output.

That difference changes the mechanics enough to warrant its own dedicated playbook, but the contrast is worth naming here:

DimensionClassic product crisisAI hallucination incident
DetectionMonitoring, alerts, customer reportsOften only a human noticing something "feels off"
ContainmentRollback, feature flag, failoverModel version pin, guardrail tightening, human-in-the-loop gate
Commander's key question"How do we stop the damage now?""How do we even confirm this is happening, and where else?"
Review focusDecision quality under time pressureDetection latency and verification gaps

The incident-command spine — one commander, defined roles, decisive containment, honest cadence, a blameless review — still applies to an AI failure. What changes is the operations lead's job: instead of a known system with a known rollback path, they're often hunting for the scope of a failure mode with no alarm bell attached. Treat that as a specialized addendum to this playbook, not a replacement for it.

Bringing Prodinja Into the Crisis Playbook

Most of what separates good crisis leadership from merely lucky crisis leadership only becomes visible afterward, when someone tries to reconstruct who knew what and when. Prodinja's Decision Journal is designed to make that reconstruction unnecessary: log the call, its confidence level, and the information behind it as the crisis unfolds, not two days later from memory. That's the single habit most incident-command playbooks assume but rarely instrument for.

Key Takeaways

  • Name one incident commander before the next crisis, not during it — authority to decide should never be improvised live under pressure.
  • Assign the four supporting roles in advance: operations, communications, scribe, and customer liaison, so nobody is negotiating who does what mid-fire.
  • Separate containment decisions from fix decisions — stop the damage first with the best available information, understand root cause second.
  • Communicate on a fixed, calm cadence even when there's nothing new to report; silence reads worse than an honest "still working on it."
  • Review decisions against what was knowable at the time, not against the outcome — resulting and hindsight bias otherwise poison the retro.
  • Log confidence levels as decisions happen, not afterward, so a blameless post-incident review can separate good process from lucky outcomes.
  • Treat AI-specific failures as a specialized addendum to this playbook — the command structure holds, but detection and containment mechanics differ.

Frequently Asked Questions

Who should be the incident commander in a product crisis?

The incident commander should be whoever can hold the full picture and make a fast, irreversible-feeling call without deferring back to the group — often a director, VP, or senior PM with cross-functional standing, not automatically the most senior person on the call or the engineer closest to the bug.

How is incident command different from a normal crisis meeting?

A normal crisis meeting gathers everyone for input and often ends without a clear decision-owner. Incident command assigns one commander and four supporting roles up front, so decisions get made by a named person on a fixed cadence instead of emerging from group consensus.

Should you blame the person who made a bad call during a crisis?

No. A sound post-incident review judges whether a decision was reasonable given the information available at the time it was made, not whether the outcome happened to work out — conflating the two punishes good judgment and rewards lucky guesses.

How often should you communicate during an active incident?

On a fixed cadence, commonly every 30 to 60 minutes, even when there's no new information to share. A short, honest "still investigating" update on schedule builds more trust than a longer, polished update sent irregularly.

Is the incident-command approach different for an AI-specific failure like hallucination?

Yes, in the details. Detection often depends on a human noticing subtly wrong output rather than an alert firing, and containment looks like model version pins or guardrails rather than a rollback — but the same commander-plus-roles structure still applies underneath.