Setting a Token Budget: The Cost Lever PMs Keep Forgetting to Own

A token budget is the maximum number of tokens—system prompt, retrieved context, and generated output combined—that a single AI feature is allowed to spend per request. Left unset, that ceiling defaults to whatever the model's context window allows, and cost becomes a function of user behavior instead of product design.

Quick answer: A token budget caps total tokens per request across prompt, retrieved context, and output. Set it deliberately as a product constraint tied to unit economics, or the model's maximum context window quietly becomes your default—and every retrieval mistake turns into a line item on next month's bill.

Why a Token Budget Is a Product Decision, Not an Infrastructure Setting

A token budget belongs to the PM because it directly sets cost per request, response latency, and answer quality—three metrics product already owns everywhere else. Treat it as an engineering default and you've outsourced a unit-economics decision to whoever wrote the retrieval query first, with nobody accountable for the trade-off.

Tokens aren't words. Anthropic and OpenAI both split text into subword chunks, so "unbelievably" might be three or four tokens and a stray Unicode character can silently cost more than a whole English sentence. AI researcher Andrej Karpathy has pointed out that tokenization itself is a frequent source of LLM quirks and cost surprises—teams that never inspect their actual token counts are budgeting blind.

Three things ride on the number you pick:

• Cost. Every provider bills per token, input and output separately, so a request's price is a direct function of what you stuffed into the context window.
• Latency. More input tokens means more time to first token; more output tokens means a longer generation. Users feel both.
• Quality. Stanford researchers Liu et al., in their widely cited "Lost in the Middle" study, found that model performance on long-context tasks degrades noticeably when the relevant information sits in the middle of a large context rather than near the start or end. A bigger budget is not automatically a better answer.

This is the same discipline the FinOps Foundation has spent years formalizing for cloud spend: cost is not solely an engineering concern once it scales with usage—it needs an owner who understands the product trade-off, not just the infrastructure bill. A token budget is FinOps for LLM calls, and the PM is the natural owner because the trade-offs are product trade-offs, not implementation details.

The difference shows up most clearly in how each approach fails:

A default isn't a decision—it's the absence of one, dressed up as a technical detail. Owning the budget means the ceiling was chosen on purpose, against a number that matters to the business, rather than inherited from whatever the SDK's example code happened to set.

Understanding this ownership question is really an extension of context engineering as a discipline distinct from prompt engineering—see the difference between context and prompt engineering for why what you feed the model, not just how you phrase the ask, is now the PM's job.

How to Allocate a Budget Across System Prompt, Retrieved Context, and Output

Split a token budget into three pools—system prompt (fixed instructions), retrieved context (variable, pulled per request), and output (the generated response)—then size each against the job the feature performs, not a round number like 8K or 32K tokens pulled from a model card.

System prompt: the fixed cost you pay on every call

The system prompt carries role definition, tone, constraints, and few-shot examples. It's the same on nearly every request, which makes it the easiest pool to audit and the first place bloat hides—instructions get added over months and nobody ever removes one.

• Cap it explicitly. If your system prompt has grown past 1,000–1,500 tokens, look for redundant instructions or examples doing the same job.
• Reuse rather than repeat. Provider-side prompt caching (both Anthropic and OpenAI offer versions of it) can cut the effective cost of a stable system prompt to a fraction of its face-value token count.

Retrieved context: the pool that actually varies

This is where document chunks, search results, prior conversation turns, and tool outputs land—and where a budget matters most, because it's the pool most exposed to unbounded growth. A feature that retrieves the top 20 matching chunks instead of the top 5 doesn't just cost more; per the Lost in the Middle findings, it can produce a worse answer.

Output: the pool you control with one parameter

Output tokens are typically the most expensive per token—both Anthropic and OpenAI price output at several times the input rate—and the one lever most teams already touch: the max_tokens parameter. Set it too high and you invite rambling answers; too low and you truncate mid-sentence.

The table below shows how the split might look across a few common AI feature types. These are illustrative starting points, not universal rules:

A support reply needs little context but should stay terse; a summarizer inverts that, spending most of its budget on the source material and returning a compact output. Mapping the split to what the context packet actually needs to contain, instead of defaulting every feature to the same number, is the allocation exercise itself.

Why round numbers like 8K or 32K are a trap

A model card advertises a maximum context window, not a recommended one. Picking 8K or 32K because it's the number on the pricing page skips the actual sizing question: what does this feature need to do its job well, at a cost the business can sustain per request? Two features on the same model can reasonably land on very different budgets.

Treat the model's maximum as a hard ceiling you're allowed to approach, never a target you default to. The gap between "what the model can hold" and "what this feature should spend" is exactly the space a PM is meant to fill in.

Why a Hard Cap Forces Better Retrieval and Exclusion Choices

A fixed budget works because scarcity forces prioritization: once "retrieve everything relevant" isn't an option, a team has to decide what actually earns a spot in the context window. That constraint routinely produces better retrieval logic than an unlimited one ever would, because someone finally has to define "relevant."

Without a cap, the default failure mode is retrieving too much on the theory that more context can only help. It can't. Beyond the added cost, the Lost in the Middle research suggests irrelevant or redundant chunks dilute the model's attention on the parts that matter—and every extra chunk raises the odds that something contradictory or stale sneaks in and gets treated as fact.

A budget cap forces four decisions a team would otherwise skip:

1. Relevance thresholds. A minimum similarity score below which a chunk doesn't make the cut, instead of "top-K regardless of score."
2. Deduplication. Collapsing near-identical chunks from different sources so the same fact doesn't eat budget twice.
3. Summarization before injection. Compressing a long document into a dense abstract rather than pasting it in full.
4. Explicit exclusion rules. Deciding what never belongs in context—outdated records, low-confidence sources, or data irrelevant to the task at hand.

That last one deserves its own discipline. Knowing what to leave out of a context packet is as consequential as knowing what to include in context—both are budget decisions in disguise. The right cut almost always traces back to the job the user is actually trying to get done: a framework like Jobs to Be Done is a sharper filter for "does this chunk matter" than a similarity score alone, because it asks what progress the user is hiring the feature to make, not just what text sits topically nearby.

Signs your retrieval is already bloated

A few tells that a feature is retrieving on autopilot rather than by design:

• The same fact appears in three or four retrieved chunks, pulled from slightly different source documents.
• Nobody can say, without checking the code, how many chunks a typical request retrieves.
• Adding a new document source to the index never triggers a conversation about whether the budget still holds.
• The retrieval step has no minimum relevance score—only a fixed "top-K" count, regardless of how weak the matches are.

Any one of these is a sign the exclusion logic is missing, not just the budget.

The Real Cost of Leaving Token Budgets Unset

An unset token budget doesn't mean "no cost"—it means the cost gets decided by whichever user, document, or edge case shows up next, instead of by product design. A single support ticket with a 40-page PDF attached, or one agentic loop re-reading its own history every turn, can become the most expensive request type in the system by accident.

Common ways this shows up in production:

• Pagination bugs in a retrieval layer that fetch far more chunks than the query needed.
• Users pasting long documents into a field with no length guard, each one billed at full context rates.
• Agentic loops that re-inject the full conversation history on every step, so cost compounds with each turn instead of staying flat.
• Debug or verbose modes left on in production, quietly appending logs or reasoning traces to every call.

To see why this matters at scale, it helps to price it out. Provider rates vary and change often, so treat the figures below as directional, not a quote—but the ratios are the point:

A few cents per request looks trivial until it's multiplied by volume. A feature running 500,000 requests a month at the frontier-tier rate above sits north of $15,000–$17,000 monthly.

That's before anyone doubles the context because a retrieval bug fetched twice as many chunks as intended. Anthropic's and OpenAI's own published pricing pages make the same case on their own: the gap between input and output rates alone is reason enough to guard output length separately from context length.

The budget question also shifts depending on where in the customer journey a request happens—a first-run onboarding flow can usually run lean, while a power-user deep-research session justifies a larger, deliberately approved ceiling. Treating every moment in the journey as needing the same budget is itself a design choice, and rarely the right one.

A Practical Framework for Setting, Testing, and Monitoring Token Budgets

Setting a token budget is a five-step exercise for a PM to run, not hand off wholesale: benchmark actual usage, size the ceiling against margin, add a hard stop with graceful degradation, watch the tail instead of the average, and revisit the number on a schedule rather than never.

1. Benchmark before you cap. Instrument current requests to log tokens per pool—system prompt, context, output—for a representative sample. You can't set an informed ceiling on a number you've never measured.
2. Size the ceiling to margin, not to the model's max. Decide what a request is allowed to cost given the feature's pricing or value, then work backward to a token ceiling, rather than defaulting to the largest context window the model supports.
3. Build a hard stop with graceful degradation. When a request would exceed budget, the feature should trim, summarize, or ask a clarifying question—not silently truncate mid-answer or throw an opaque error.
4. Monitor p95 and p99, not the average. Mean token usage hides the handful of requests doing 10x the typical spend; those are exactly the ones a budget exists to catch.
5. Revisit quarterly. Model prices change, usage patterns drift, and a budget set for last year's document sizes may already be stale for this year's.

A budget that's never been tested against a hard-stop scenario isn't a budget—it's a suggestion.

Where Token Budgets Fit in a Context Engineering Workflow

A token budget is not a side note to context engineering—it's one of the required fields in a complete context packet, alongside the system prompt, the retrieved sources, and the exclusion rules. A packet with everything else defined but no budget is still an unbounded-cost design, just a well-organized one.

This is the gap Prodinja's Context Engineering tool is built to catch: it flags an unset token budget the same way it flags a missing source or an undefined exclusion rule, treating the budget as a required part of a complete context packet rather than an optional afterthought. The point isn't automation magic—it's making the omission visible before it ships, the same way a linter catches a missing null check.

Whether or not a tool flags it for you, the habit is the same: don't consider a context packet finished until the budget line has a number in it, not a blank.

Key Takeaways

• A token budget is a unit-economics decision, not an infrastructure default—it sets cost, latency, and answer quality per request, all metrics product already owns.
• Split the budget into three pools—system prompt, retrieved context, and output—and size each against the job the feature does, not a round number from a model card.
• Scarcity improves retrieval. A hard cap forces relevance thresholds, deduplication, and explicit exclusion rules that an unlimited budget lets a team skip.
• Unset budgets fail expensively and silently, through pagination bugs, oversized user input, or agentic loops that re-read their own history every turn.
• Monitor the tail, not the average. The requests that blow past a sane budget are exactly the ones a budget exists to catch.
• Budgets should vary by journey stage and feature, not default to one number applied everywhere a model gets called.