When Your Model Provider Goes Down: Fallbacks, Routing, and Graceful Degradation

Design for provider failure the way you'd design for any dependency you don't control: assume it will go down, then decide in advance what "up" means for your product when it does. A fallback ladder—retry, secondary model, degraded mode, cached response—lets you trade quality for availability deliberately instead of discovering the trade-off during an incident.

Quick Answer: Build a tiered fallback ladder (retry with backoff → secondary model → rule-based degraded mode → cached response) and pick, in advance, the quality floor below which "degraded" becomes worse than "down." Route by health signals, not by hope.

Why Your Uptime Is Actually Your Vendor's Uptime

Your AI feature's availability ceiling is set by the weakest link in a chain you mostly don't operate: the model provider's inference API. No amount of caching, retry logic, or clever UX raises your uptime above whatever OpenAI, Anthropic, or Google is delivering that day, unless you've built a path around them.

This isn't hypothetical. OpenAI's status history shows recurring multi-hour degradations and outages across 2023-2025, including a widely-reported multi-hour ChatGPT and API outage in June 2025 tied to infrastructure changes. Anthropic's status page has similarly logged elevated error rates and partial outages during high-demand periods. These aren't indictments of any one vendor—they're evidence that inference at scale, across every major provider, has an availability profile closer to a utility with rolling brownouts than to a five-nines database.

The reliability engineering discipline has a name for designing around this: the Site Reliability Engineering (SRE) practice popularized by Google treats every external dependency as a probabilistic failure source with its own error budget, not a binary up/down assumption. Applied to LLMs, that means your product's reliability is a function you design, not a number you inherit.

The Dependency You Didn't Choose to Own

Most teams adopt a single model provider for good reasons—simpler prompting, one API to learn, predictable billing. But that choice quietly makes your application's ceiling identical to theirs. If your provider has 99.5% monthly uptime, your AI feature cannot exceed 99.5% uptime unless something else picks up the slack.

Two changes are worth stating plainly:

• Your SLA commitments to customers, if you make any around AI features, are only as credible as your fallback plan—not your primary vendor's contract.
• Your incident response runbook needs a branch for "the model is down" that looks nothing like "the database is down"—the recovery isn't a failover to a replica, it's a routing decision with a quality trade-off attached.

Building the Fallback Ladder: Four Rungs, Four Trade-Offs

A fallback ladder is a sequence of progressively cheaper, progressively lower-quality responses your system falls back to as the primary path degrades—each rung should be a deliberate design decision, not an afterthought bolted on after the first outage. Most mature implementations use four rungs, escalating in speed of activation and descending in output fidelity.

Rung 1: Retry with backoff absorbs transient blips—rate limits, brief 5xx spikes, network jitter—without the user ever noticing. Use exponential backoff with jitter (a pattern documented extensively in AWS's and Google Cloud's own reliability guidance) so a wave of simultaneous retries doesn't itself become a self-inflicted traffic spike against a recovering provider.

Rung 2: Secondary model or provider is where most of the architectural work lives. This requires your application to be model-agnostic at the prompt layer—no provider-specific formatting baked into business logic—so a request can be replayed against Anthropic, OpenAI, or an open-weight model served elsewhere without a rewrite. Expect behavioral drift: a secondary model may answer the same prompt with a different tone, length, or occasionally a different factual take.

When Retry Isn't Enough

Retry logic only works if the failure is actually transient. A full regional outage, a deprecated model endpoint, or a provider-side incident lasting hours will exhaust your retry budget and burn latency your users can feel. Multi-model routing exists precisely for the failures retry can't fix.

• Circuit breakers: after N consecutive failures against a provider, stop sending traffic there for a cooldown window rather than retrying into a known-down endpoint.
• Health checks: a lightweight synthetic request every 30-60 seconds tells you a provider has recovered before your real traffic finds out the hard way.
• Cost-aware routing: your secondary model is often more expensive or slower per token—route only the traffic that needs it, not everything, once the primary recovers.

Routing Logic: Deciding Who Handles the Request

A routing layer decides, per request, which model or fallback rung handles it—based on real-time health signals (latency, error rate, timeout count) rather than a static assumption that the primary provider is always available. Treat routing as its own component with its own monitoring, not a side effect of a try/catch block.

The three signals worth instrumenting, in rough order of how fast they should trigger a route change:

1. Error rate over a rolling window (e.g., >5% of requests to a provider failing in the last 60 seconds)—triggers fastest, since a spike usually means something is actively broken.
2. P95/P99 latency creeping past an acceptable threshold—slower to trigger, but catches "technically up, functionally unusable" degradation that pure error rate misses.
3. Explicit provider status signals—status page APIs or webhook feeds, where available, as a corroborating signal rather than the sole trigger, since these often lag real user-facing impact.

This connects directly to the observability foundation any AI product needs before an incident hits, not during one—see the complete guide to LLMOps observability for the metrics layer that should already exist under this routing logic, and the three dashboards worth building on day one for what to have visible before you need it.

Routing Isn't Just Failover—It's Also Cost and Latency Arbitrage

Once you've built multi-provider routing for resilience, the same infrastructure pays for itself in ways unrelated to outages. Teams route cheap, low-stakes requests to a smaller model and reserve the frontier model for requests that actually need its reasoning depth—an optimization that only exists because the failover plumbing was already there.

Building the routing layer for resilience and getting cost/latency optimization as a side effect is one of the few "eat your vegetables" investments in AI infrastructure that pays back even in a quarter with zero outages.

Graceful Degradation: When Degraded-But-Up Beats Down

Graceful degradation means your product keeps functioning—at reduced quality or reduced scope—rather than failing outright when the model is unavailable, and the decision of when that trade-off is worth making has to be made before the outage, not during it. Not every AI feature deserves a degraded mode; some genuinely should show an honest error instead of a worse answer dressed up as a real one.

The deciding question is whether a lower-quality automated answer is more harmful than no answer at all. Two examples on opposite sides:

• Worth degrading: an AI-powered search summarizer falls back to keyword-matched excerpts instead of a generated summary. Users get something useful; nobody's misled into thinking it's a synthesized answer.
• Not worth degrading: an AI feature drafting legal or medical guidance falls back to a generic template answer that looks equally confident and equally specific. Here, "down" with a clear message is safer than a plausible-sounding wrong answer.

Jakob Nielsen's usability heuristics—specifically "visibility of system status" and "error prevention"—are directly relevant here: a degraded response that doesn't disclose it's degraded is a status-visibility violation with real trust cost, and it doesn't survive contact with a discerning user for long. Always signal degraded mode; don't quietly swap the engine and hope nobody notices the drop.

Worked Example: A Two-Hour Provider Outage

Consider a customer support AI assistant, midday, at peak traffic, when its primary provider's API starts returning 503s.

The team that built this ladder in advance spent two hours at slightly higher cost and slightly lower quality. The team without one spent two hours either fully down or serving errors—and likely fielding support tickets that took longer to resolve than the outage itself.

Testing Your Fallback Ladder Before You Need It

A fallback ladder that's never been exercised is a hypothesis, not a system—the only way to know retries, routing, and degraded mode actually work under real failure conditions is to simulate the failure deliberately, on a schedule, before a real provider outage forces the first live test.

Borrow directly from Netflix's Chaos Engineering practice (originated with Chaos Monkey and formalized in the Principles of Chaos Engineering): inject synthetic provider failures—simulated timeouts, forced 500s, artificially throttled rate limits—against a staging or canary slice of production traffic on a recurring basis.

1. Simulate a primary-provider outage and confirm the circuit breaker opens within your target window, not just eventually.
2. Verify the secondary model actually produces usable output for your top request types—don't assume parity you haven't tested.
3. Confirm degraded-mode messaging renders correctly—a silent quality drop is the failure mode you're trying to avoid, not just downtime itself.
4. Time the full round trip from failure detection to user-visible fallback; this number belongs on a dashboard, watched the same way you'd watch quality drift that doesn't throw a stack trace.

Every incident, real or simulated, should feed back into your eval suite so the same failure mode gets caught earlier next time—the same discipline covered in closing the loop from failures to eval cases applies just as directly to infrastructure failures as it does to quality regressions.

Deciding If Multi-Provider Resilience Is Worth Building

Not every product needs a four-rung ladder with two live providers on day one—the cost of building and maintaining multi-provider resilience should be weighed against the actual risk it buys down, which scales with how much of your product's core value depends on the model being reachable at any given moment.

A support chatbot that's a nice-to-have on a support page carries different stakes than an AI feature that's the entire product. This is exactly the kind of trade-off RICE and Kano prioritization, as modeled in Prodinja's prioritization studio, is designed to help you weigh: reach and impact scale with how central the AI path is to your core flows, while effort captures the real cost of a second provider integration, a routing layer, and ongoing dual-vendor testing—so the resilience investment gets sized against the outage risk it actually offsets, not built reflexively because a competitor mentioned it.

Key Takeaways

• Your uptime is capped by your model vendor's—design a fallback ladder deliberately rather than discovering your ceiling during an incident.
• Four rungs cover most cases: retry with backoff, secondary model/provider, rule-based degraded mode, and cached response, each with a distinct quality-for-availability trade-off.
• Route on real signals—error rate, P95/P99 latency, and provider status—not on the assumption that the primary provider is always healthy.
• Degraded-but-up beats down only when the degraded answer is honestly disclosed; a confident-sounding wrong answer can be worse than a clear outage message.
• Test the ladder before you need it—chaos-engineering-style simulated outages are the only reliable way to know retries and routing actually work.
• Weigh multi-provider investment against the risk it buys down—not every product needs two live vendors, and a framework like RICE/Kano can make that trade-off explicit instead of reflexive.

Frequently Asked Questions

What should I do when my LLM provider's API goes down?

Route traffic through your fallback ladder in order: retry with exponential backoff for transient errors, then failover to a secondary model or provider, then drop to a rule-based degraded mode, then serve cached responses if available. Which rung activates depends on error rate and latency thresholds you set in advance, not real-time judgment calls during the incident.

How many model providers should I integrate for resilience?

Two is usually the practical minimum for real redundancy—a primary and a secondary you can fail over to—since a single provider offers no failover path at all. Whether a third is worth it depends on how central the AI path is to your product; use a framework like RICE to weigh the added integration and maintenance cost against the incremental risk it removes.

Is it better to show an error or a degraded AI response during an outage?

It depends on whether a lower-quality automated answer is more harmful than no answer—show a degraded response, clearly disclosed, for use cases like search summarization where "something useful" beats nothing; show an honest error for high-stakes use cases like legal or medical guidance where a plausible-but-wrong answer causes more harm than a clear "unavailable" message.

How do I test multi-model routing without waiting for a real outage?

Use chaos-engineering techniques—inject synthetic failures like forced timeouts, simulated 500 errors, or artificial rate-limit throttling against a staging or canary slice of traffic on a recurring schedule. This is the same practice Netflix formalized with Chaos Monkey, applied to your model provider dependency instead of infrastructure services.

Does multi-provider routing help with anything besides outages?

Yes—once routing infrastructure exists for resilience, teams commonly reuse it for cost and latency arbitrage, sending simpler requests to smaller or cheaper models and reserving frontier models for requests that need the extra reasoning depth. It's one of the rare reliability investments that pays back even during quarters with zero outages.