Sending every request to your most capable model is the default, and it is a margin leak. Model routing sends easy, low-stakes requests to a cheaper model and reserves the expensive one for genuinely hard or high-stakes cases, using signals like query complexity, confidence scores, and downstream stakes. Done well, it holds quality flat while cutting spend by half or more.
Quick Answer: Route requests to the cheapest model that clears a pre-defined quality bar for that request's difficulty and stakes — not to your best model by default. Measure the tradeoff live on a cost-vs-quality dashboard, and set the "good enough" bar with the same rigor you'd use for a human SLA.
Most AI products launched with a single model wired to every request, because a single model was the fastest way to ship. That default persists long after usage patterns make it wasteful: a support-ticket classifier, a first-draft summarizer, and a contract-clause extractor all end up calling the same frontier model, even though only one of those tasks reliably needs it. Model routing is the fix, and it's a platform PM problem, not just an infra one — the routing policy is a product decision about which requests deserve premium capability.
Why Uniform Routing Is a Margin Leak
Routing every request to your best model treats a support-ticket triage as equally demanding as a legal-clause extraction, when the two tasks have wildly different difficulty and error tolerance. The result is systematic overpayment: you're buying frontier-model reasoning for requests that a smaller model answers identically, at a fraction of the per-token cost.
The economics compound with scale. A frontier model can run 5-15x the per-token cost of a mid-tier model from the same provider family, and most production traffic is not evenly distributed across difficulty — it skews toward simple, repetitive requests. Anthropic's and OpenAI's own published pricing tiers make this gap explicit; it isn't a rounding error, it's the difference between a viable unit economics model and one that erodes gross margin as usage grows.
Three forces make uniform routing worse over time, not better:
- Usage grows faster than revenue per request. Free-tier and low-intent usage often scales fastest, and it's exactly the traffic that doesn't need your best model.
- Model prices diverge, not converge. Frontier capability keeps getting more expensive at the top even as commodity tiers get cheaper — the spread between "best" and "good enough" widens.
- Feature creep adds calls, not just complexity. Every new AI feature is another call site defaulting to whatever model the last one used, compounding the leak silently.
This is the same blind spot covered in the complete guide to LLMOps observability — cost is a first-class signal, not a footnote to quality. If you're only watching quality metrics, you won't see the leak until finance asks about it.
Routing Signals: What Actually Predicts Difficulty
A routing decision needs a difficulty estimate made before the expensive model runs, built from three signal families — query complexity, model confidence, and downstream stakes. None of these signals is sufficient alone; production routers combine at least two.
Query Complexity Signals
Complexity signals are the cheapest to compute because they don't require calling any model first. They estimate difficulty from properties of the input itself.
- Input length and structure: a one-line factual question is a different difficulty class than a multi-paragraph request with embedded constraints.
- Task-type classification: a lightweight classifier (often a small, cheap model itself) tags the request as extraction, generation, reasoning, or creative — each with a different baseline difficulty.
- Novelty against cached patterns: requests matching a known-easy pattern (a FAQ-shaped question, a templated field extraction) route down by default.
- Domain specificity: requests touching a narrow, well-covered domain are usually easier than open-ended cross-domain reasoning.
Confidence and Self-Consistency Signals
Confidence signals require running something — often the cheap model itself — and reading its own uncertainty as a routing input rather than a final answer.
- Token-level log-probabilities: low average confidence on the cheap model's own output is a strong escalation trigger.
- Self-consistency checks: sampling the cheap model twice and comparing answers; disagreement signals the request needs a stronger model.
- Calibrated confidence scores: some providers expose confidence or verbosity settings you can read as a proxy, though calibration quality varies by model family and should be validated against your own eval set before you trust it.
Stakes Signals
Stakes signals come from your product's own context, not the model — they answer "what happens if this response is wrong?"
| Stakes tier | Example request type | Routing default |
|---|---|---|
| Low stakes | Draft summary, internal note, autocomplete | Cheapest model that passes eval |
| Medium stakes | Customer-facing reply, first-pass classification | Mid-tier model, escalate on low confidence |
| High stakes | Contract clause, medical/financial guidance, irreversible action | Best available model, human review gate |
A request that's easy and low-stakes is the clearest case for the cheap model. A request that's easy but high-stakes still deserves scrutiny — difficulty and stakes are independent axes, and conflating them is the most common routing mistake. This mirrors the emotional-stakes mapping in a customer journey analysis: the moments that matter most aren't always the moments that are hardest to serve.
The Cost-Quality Dashboard: Watching the Tradeoff Live
A routing policy without a live dashboard is a bet you can't see the odds on. The dashboard needs exactly two paired numbers, tracked together over time and by segment: cost saved versus quality delta, never one without the other.
What Belongs on the Dashboard
- Cost per request, blended and by route — the actual spend, not a theoretical estimate, split by which model handled each request.
- Quality delta versus an all-frontier baseline — measured on the same eval set the frontier model was scored against, not a separate easier benchmark.
- Escalation rate — the percentage of requests the cheap model bounced up to the expensive one, and whether that rate is drifting.
- Route distribution over time — what fraction of traffic lands on each tier, watched for silent drift as usage patterns shift.
- Cost saved per 1,000 requests — the number that actually goes in a budget conversation.
This is the same instinct behind having three dashboards on day one for any AI product: usage, quality, and cost, viewed together rather than in separate tools nobody cross-references. A routing policy specifically needs cost and quality on one screen, because the entire point of routing is the tradeoff between them — split them into different tools and the tradeoff becomes invisible.
Reading the Dashboard Without Fooling Yourself
Quality delta near zero with a large cost drop is the target state, but it's only trustworthy if the eval set covers the same difficulty distribution as production traffic. A win on an easy-skewed eval set won't hold once real traffic's harder tail routes down too.
Watch for route drift: a routing rule tuned for last quarter's traffic mix quietly serves more hard requests to the cheap tier as your product's usage evolves, because nobody re-validated the split. Segment the dashboard by request type and revisit the split monthly, not just when quality complaints arrive.
A Worked Example: 70% of Traffic, Half the Spend
Consider a support-assistant workload where a platform PM wants to test whether a cheaper model can absorb the bulk of traffic without a visible quality drop. The setup: a difficulty classifier tags each incoming ticket, low-and-medium-difficulty tickets route to a cheaper model, and anything flagged high-difficulty or low-confidence escalates to the frontier model.
| Metric | All-frontier baseline | Routed (70% to cheap model) |
|---|---|---|
| Requests to frontier model | 100% | ~30% |
| Requests to cheaper model | 0% | ~70% |
| Eval-set quality score | Baseline (100%) | Held flat, within noise |
| Blended cost per 1,000 requests | Baseline (100%) | ~50% of baseline |
| Escalation rate (cheap → frontier) | n/a | Monitored, tuned to stay low |
The mechanism that makes this work isn't the cheap model getting smarter — it's that most of the 70% never needed frontier-level reasoning in the first place. The escalation path is what protects quality: any request the cheap model handles with low confidence still reaches the frontier model, so the "flat quality" outcome depends entirely on the escalation trigger being well-calibrated, not on hoping the cheap model guesses right.
This is a directional illustration of the mechanism, not a result Prodinja has measured on live customer traffic — the platform is a prototype today, and the specific 70%/50% split you'd see depends entirely on your own traffic mix and difficulty distribution. Treat it as the shape of the outcome routing is designed to produce, and validate the actual numbers against your own eval set before reporting them upward.
Setting a Defensible "Good Enough" Bar
Routing only works if "good enough" is a specific, written, testable bar — not a vibe someone eyeballs after the fact. Without that bar, every routing decision becomes a negotiation, and every quality regression becomes a blame exercise between whoever picked the cheap model and whoever approved it.
A defensible bar needs three properties: it's specific (a numeric or rubric threshold, not "seems fine"), it's tested against real failure modes (drawn from actual production misses, not hypothetical ones), and it's owned (someone signs off on where the line sits, and that decision is visible to whoever audits spend later). This is precisely the gap the rubric structure in Prodinja's Evals concept is designed to make explicit: instead of an implicit, undocumented sense of "good enough" living in one engineer's head, the rubric states the pass bar per task type in a form that survives the person who wrote it leaving the team.
Without that explicit bar, quality drift becomes invisible until a customer notices — the exact failure mode covered in why quality drift doesn't throw a stack trace. A routing policy is a quality policy wearing a cost hat; treat the threshold with the same rigor you'd apply to any other production SLA, and revisit it every time you change which model sits at either end of the route.
Key Takeaways
- Uniform routing is a hidden cost, not a safe default — sending every request to your best model overpays for the majority of traffic that doesn't need frontier-level reasoning.
- Combine at least two signal families — query complexity, model confidence, and downstream stakes each catch different failure modes alone.
- Stakes and difficulty are independent axes — an easy, high-stakes request still deserves scrutiny; don't route purely on perceived difficulty.
- Track cost and quality on one dashboard, never separately — a cost win only means something next to the quality delta it cost you.
- Escalation rate is your early-warning signal — a rising escalation rate on the cheap tier often means traffic composition is shifting, not that the cheap model got worse.
- "Good enough" must be a written, owned, testable bar — an explicit rubric, not an implicit judgment call, is what makes a routing policy defensible when someone audits it later.
- Revisit the routing split on a schedule, not just on complaints — traffic mix drifts, and a policy tuned for last quarter's distribution can quietly under-serve this quarter's harder tail.
Frequently Asked Questions
What is model routing in LLM applications?
Model routing is the practice of sending each request to whichever model is cheapest while still meeting a pre-defined quality bar for that request's difficulty and stakes, rather than sending all traffic to a single model by default. It requires a difficulty or confidence signal computed before the expensive model runs.
How do you measure the cost-quality tradeoff for LLM routing?
Measure cost per request and quality delta against an all-frontier baseline on the same eval set, tracked together over time rather than in separate tools. A dashboard showing cost saved next to quality lost — not either alone — is what makes the tradeoff auditable.
Is a cheaper model always lower quality than a frontier model?
Not for every request — a cheaper model matches frontier quality on requests within its competence, which is often the majority of production traffic. The gap only shows up on genuinely hard or ambiguous requests, which is exactly what a difficulty-based router is meant to detect and escalate.
How much can model routing actually save on LLM costs?
Savings scale with how skewed your traffic is toward easy requests and how wide the price gap is between your cheap and frontier models — often a 5-15x per-token difference within the same provider family. The realistic range varies by workload; validate your own split against your own eval set rather than assuming a fixed percentage.
What's the risk of routing too aggressively to a cheap model?
The main risk is silent quality drift on requests the router misclassifies as easy, since a routing failure doesn't throw an error — it just returns a worse answer. Mitigate it with a low-confidence escalation path and a written quality bar, then monitor escalation rate as an early-warning signal.