Model Routing: Cheap-and-Fast by Default, Expensive-and-Smart on Demand

Model routing treats LLM selection as a runtime policy decision, not a fixed architecture choice: send most requests to a small, cheap, fast model, and escalate only the queries that actually need frontier-level reasoning. Done well, it can cut blended inference cost substantially without users noticing any drop in quality.

Model routing is a policy layer, not a model choice: classify difficulty, default to a small/fast model, and escalate only when confidence is low, complexity signals fire, or a validation check fails. Most production query mixes are lopsided enough that an 80/20 cheap/expensive split covers the bulk of the savings.

Why Your Biggest Model Is the Wrong Default

Most production LLM traffic is easy: short factual lookups, formatting requests, simple classification, or restating something already in context. Routing every one of those through your most expensive, highest-latency model wastes money and adds latency for no quality benefit the user can perceive.

Teams reach for the flagship model by default because it removes a decision. Nobody gets blamed for using frontier-tier reasoning on every request — it feels like the safe choice. But that default has a cost curve, and it compounds: every simple classification, extraction, or formatting call inherits the price and latency of your most capable, and usually slowest, model.

Why bigger models cost more per token and add more latency comes down to parameter count and inference compute, which is worth understanding at a lower level if you haven't already covered how LLMs actually generate a response. The short version: a model tier is a bet placed once, at request time, on how hard the problem might turn out to be — and most requests aren't that hard.

Query difficulty in production systems tends to follow a lopsided distribution: a large majority of requests are routine, and a small minority require genuine multi-step reasoning, ambiguous judgment calls, or synthesis across long context. Treating every request as if it belongs to that minority is the single most common unforced error in LLM feature cost design.

The Router Is a Policy Layer, Not a Model

A router is not another model in your stack — it is a decision function that sits in front of your models and decides, per request, which one gets called. That function can be as simple as a rules-based classifier or as involved as a small trained model predicting expected quality per tier.

In practice, teams build routers three ways, in increasing order of sophistication:

1. Rule-based routing — a lightweight classifier or even a regex/keyword heuristic sorts requests by task type (for example, "summarize this" versus "debug this code") and maps task types to a pre-assigned tier. Cheap to build, brittle as your feature set grows.
2. Cascade routing — every request starts at the cheap model; a scoring function inspects its own output (confidence, format validity, length) and decides whether to retry with a stronger model. This is the architecture behind Stanford's FrugalGPT research (Chen, Zaharia & Zou, 2023), which showed that LLM cascades can preserve most of a flagship model's accuracy while cutting the cost of the overall pipeline substantially.
3. Learned routing — a small classifier is trained on labeled data, often human preference judgments, to predict, before any model runs, which tier is likely to satisfy the request. RouteLLM (LMSYS / UC Berkeley, 2024) took this approach, training routers on Chatbot Arena preference data and reporting that a well-tuned router could approach flagship-model quality while calling the expensive model for only a minority of requests.

All three are descendants of the same idea that powers Mixture-of-Experts models internally. Shazeer et al.'s 2017 sparsely-gated routing layer showed that a network deciding which sub-component handles a token beats forcing every token through the same dense path. Model routing just moves that decision from inside the model to the API layer, where a PM can actually see it, own it, and tune it.

The Escalation Rule: What Should Trigger the Expensive Model

The most reliable escalation rule combines two signal families: the cheap model's own uncertainty (confidence score, hedging language, retry count) and the structural complexity of the request itself (output length, number of steps, schema validation failure). Escalate when either crosses a pre-set threshold — not when a human guesses the request "feels hard."

Confidence-based signals

Most model APIs expose, or can be coaxed into providing, some proxy for confidence:

• Token-level log-probabilities — a low average log-probability on generated tokens correlates with the model being unsure, especially on factual or numeric answers.
• Self-rated confidence prompts — asking the cheap model to append a 1-10 confidence score to its own answer is a crude but usable signal. It's worth being skeptical of it at face value, per the calibration research from Guo et al. (2017), which showed modern neural networks are often systematically over- or under-confident and need explicit calibration before their scores can be trusted.
• Consistency checks — sample the cheap model twice at low marginal cost and compare the two answers; disagreement is itself an escalation signal, a cheap cousin of self-consistency sampling.

Complexity signals

Independent of confidence, some requests are just structurally more demanding:

• Output length exceeds an expected ceiling — the model is padding, hedging, or reasoning through something long-form.
• The request requires multi-hop reasoning across more context than a single pass reliably handles.
• A downstream validation check fails — JSON doesn't parse, generated code doesn't compile, a required field is missing.
• The user explicitly rejects the first answer — a thumbs-down, a "try again," or a follow-up that just restates the question.

A concrete rule

A workable starting rule, in pseudocode, looks like this:

escalate = (confidence_score < 0.6) OR (output_tokens > 400) OR (schema_valid == False) OR (retry_count >= 1)

Tune the thresholds against your own held-out set rather than copying these defaults — the right confidence cutoff for a support-ticket classifier is not the right cutoff for a code-generation assistant. What matters is that the rule is explicit, versioned, and testable, not a vibe an engineer encoded in a prompt once and never revisited.

The Cost-Mix Math: What an 80/20 Split Actually Buys

An 80/20 traffic split — 80% of requests handled by a small, fast model and 20% escalated to a large, smart model — is a realistic starting assumption for most production LLM features, and it changes your blended cost dramatically compared to routing everything to the expensive tier. The table below shows why.

At these illustrative ratios, routing brings blended cost to roughly a quarter of the all-flagship baseline. The exact multiplier between your smallest and largest model's per-token price will vary by provider and shift over time, but a 10-30x gap between a small and a frontier tier is common across major model families, so run this table with your own current pricing before presenting it in a roadmap review.

Before you even reach the router, some fraction of that 80% doesn't need a live model call at all. If the same or a near-duplicate question has already been answered, semantic caching can intercept it for a fraction of even the cheap model's cost, and it pairs naturally with routing rather than competing with it — cache first, route what's left.

The 80/20 split itself is a starting assumption, not a universal constant. Your own ratio only becomes trustworthy once you've measured it against a sample of real production requests, which is exactly the kind of instrumented listening product discovery is meant to produce, rather than a top-down guess pulled from a benchmark paper.

Where Routing Breaks: Failure Modes and Guardrails

Routing fails quietly, which makes it dangerous: a misclassified hard request gets a shallow answer instead of an error, and nobody notices until a user complains or a metric drifts. The fix is treating router decisions as a monitored, evaluated system, not a one-time cutoff you set and forget.

A router that fails silently is worse than no router at all — you've added a system that can be confidently wrong without anyone noticing.

Silent degradation is the core risk

A misrouted request rarely crashes; it just produces a worse answer with full confidence. That is a failure mode, and it deserves the same deliberate design as any other failure state — a wrong-tier response should get the same graceful-degradation thinking laid out in the UX of failure: signal uncertainty to the user rather than presenting a shallow answer as authoritative.

The cascade latency tax

Cascade routing calls the cheap model first and only escalates after evaluating its output, which means every escalated request pays for two round trips instead of one. If you can detect obviously-hard requests upfront — very long input, a task type you already know always escalates — route them directly to the expensive model and skip the cheap-model detour. Otherwise you're trading cost for latency on exactly the requests where users are least patient.

Drift in both directions

Providers update model behavior behind stable API names, query distributions shift as your product finds new use cases, and a threshold tuned in one quarter can be miscalibrated by the next. Multi-step agentic workflows make this worse, because every tool call or reasoning step is its own routing decision, so drift compounds across a chain instead of showing up once. Treat your router the way you'd treat any model in production: with a held-out eval set, scheduled re-checks, and a changelog.

Calibration bias across segments

If your confidence signal is systematically noisier for certain inputs — non-native phrasing, unusual formatting, domain jargon — you can end up silently over- or under-escalating for reasons that have nothing to do with actual difficulty. Audit escalation rates by input segment, not just in aggregate.

Choosing the Axis: When Routing Is (and Isn't) the Right Architecture

Routing is the right architecture when cost and latency both matter more than squeezing out marginal quality on easy requests. It is the wrong fix if your real problem is quality on a narrow set of hard requests, where a better prompt or a single stronger model would be simpler and just as effective.

The three axes that matter

Every LLM feature decision, routing included, is ultimately a trade-off across three axes: cost, latency, and quality. Routing is a strong answer only when at least two of those three are under real pressure at the same time. If quality is the only constraint that matters — a legal-document summarizer where errors are expensive — routing adds complexity without addressing the actual risk. If cost and latency both matter and quality has headroom, routing is close to a textbook fit.

A quick self-check

Ask three questions before building a router:

1. Do we actually have a lopsided query mix? If most of your traffic is already hard, routing saves little — you'd mostly be paying for the expensive model anyway.
2. Can we detect difficulty before or immediately after generation? If not, none of the escalation signals covered above will be reliable enough to trust.
3. Is quality on the hard 20% non-negotiable? If yes, make sure your escalation rule errs toward over-escalating, not under-escalating — a false economy here costs more in trust than it saves in tokens.

This is exactly the kind of multi-axis reasoning that structured trade-off analysis is built for — routing is one architectural answer among several, and it's only the right one once you know which axis is actually under the most pressure. Prodinja's Trade-off Triangle, part of the same prototype, is designed to walk you through mapping cost, latency, and quality against each other for a specific feature, and to point you toward a routing-first architecture when cost and latency both dominate the conversation, rather than defaulting to "just use the best model" out of habit.

Key Takeaways

• Routing is a policy layer, not a model — it decides which model handles a request, and belongs in your architecture diagram as its own component, not as a footnote inside a prompt.
• Escalate on evidence, not vibes — combine a confidence signal (log-probs, self-rated score, consistency check) with a complexity signal (output length, validation failure, multi-hop reasoning) into one explicit, versioned rule.
• An 80/20 split is a reasonable starting assumption, but measure your own query distribution before trusting it — the savings math only holds if the split reflects reality.
• Cascades trade cost for latency on escalated requests; detect obviously-hard requests upfront when you can, and skip the cheap-model detour.
• Silent degradation is the real risk, not outright failure — a misrouted request looks confident even when it's wrong, so audit escalation rates by segment, not just in aggregate.
• Routing pairs with, but doesn't replace, semantic caching — caching intercepts repeat questions before any model runs; routing decides which model runs when caching doesn't apply.
• Routing earns its complexity when cost and latency both matter alongside quality — if quality is your only real constraint, a stronger single model or a better prompt is usually simpler.