Context budgeting means treating an agent's context window as scarce, finite real estate and deciding deliberately what belongs in it: system instructions, current task state, retrieved facts, and history, in that priority order. Everything else should be fetched on demand rather than loaded upfront. The discipline prevents the two failure modes that plague agent design — starving the model of what it needs, and drowning it in what it doesn't.
Quick Answer: Rank what goes into an agent's context by priority — instructions first, task state second, retrieved facts third, history last — and default to retrieval-on-demand for anything not needed on every single turn. Stuffing the window with "just in case" material degrades reasoning quality and inflates cost per call.
Why Context Is a Budget, Not a Warehouse
A context window behaves like a budget because every token you spend on one thing is a token you can't spend on another, and spending more doesn't guarantee better output. Even as context windows have grown into the hundreds of thousands of tokens, model quality on long-context tasks doesn't scale linearly with the room available — it degrades unevenly as the window fills.
Researchers at Stanford, UC Berkeley, and Samaya AI documented this directly in the "Lost in the Middle" study (Liu et al., 2023), which found that language models retrieve information from the start and end of a long context far more reliably than from the middle. Performance on multi-document question answering dropped noticeably once the relevant fact was buried in the middle of a long input, even when the model had technically "seen" it. Anthropic's own guidance on long-context prompting echoes this: placement and structure inside a large context window still matter, because attention isn't uniform across it.
The budget framing forces a question a warehouse framing never asks: does this specific token justify its position, or is it there because it was easier to include than to leave out? Three practical consequences follow.
- More context isn't automatically more accurate. Irrelevant or redundant material dilutes the signal the model has to find and act on, even if it never technically causes an error.
- Cost tracks context linearly, output quality doesn't. You pay per input token regardless of whether that token helped, so an inflated context is dead weight that keeps you paying.
- Attention degrades with distance from the edges. Facts placed mid-window are the least reliably retrieved, which means where you put something matters as much as whether you include it.
The Four-Tier Priority Stack
The four-tier stack — instructions, task state, retrieved facts, history — gives you a default ordering for what earns space when the window gets tight, and it maps cleanly onto how agent frameworks already structure a prompt. Treat lower tiers as negotiable and higher tiers as close to fixed.
| Tier | What it contains | Typical share of budget | Compression strategy when tight |
|---|---|---|---|
| 1. Instructions | System prompt, role, constraints, output format | 5-15% | Rarely compress; this is the cheapest tier to get wrong |
| 2. Task state | Current goal, in-progress plan, tool outputs from this turn | 15-30% | Summarize completed sub-steps, keep only the active one in full |
| 3. Retrieved facts | Documents, code, records fetched for this specific step | 30-60% | Retrieve narrower slices; rerank before inserting |
| 4. History | Prior turns, past tool calls, conversation so far | Remainder | Roll up into a running summary; keep only the last N raw turns |
Instructions: small, fixed, load-bearing
Instructions should be the smallest tier by token count and the most stable across a run. They set the agent's role, boundaries, and output contract, and they need to survive intact through every subsequent step of a long task. A bloated system prompt that tries to anticipate every edge case usually backfires — it competes with task-specific content for the model's attention and often gets partially ignored anyway.
This is exactly the territory covered in a five-part agent spec structure: goal, constraints, tools, examples, and output format, kept tight enough that nothing in it is optional filler. If an instruction only applies to a rare edge case, it belongs in retrieved facts or a conditional tool description, not baked permanently into every single call.
Task state: the working plan, not the archive
Task state is what the agent is doing right now — its current sub-goal, the plan it's executing against, and the output of the most recent tool call. This tier changes turn to turn and should be re-written, not accumulated. An agent three steps into a five-step plan doesn't need the verbose reasoning from step one preserved verbatim; it needs a one-line summary of what was decided and why.
A common anti-pattern is letting task state grow by accretion — appending each step's full output to the previous ones until it functions as an ever-growing log. That's a symptom of not having a clear goal for the step, a problem addressed in how to write an agent goal without drift: a narrowly scoped goal naturally bounds how much state needs to persist to serve it.
Retrieved facts: pull, don't preload
Retrieved facts are the material fetched specifically to serve the current step — a code diff, a style guide section, a customer record — and this is where the stuffing-versus-retrieval trade-off is sharpest. The instinct to "just include the whole file" or "just paste in the whole knowledge base" feels safer than retrieval, but it's rarely correct.
History: compress aggressively, keep the tail literal
History is the lowest-priority tier and the one most safely compressed, because an agent generally needs to know what happened, not relive it verbatim. A rolling summary of completed turns, paired with the last one or two turns kept in full detail, preserves continuity without the linear cost growth of keeping every turn raw. Long-running agents that skip this step are the ones that eventually hit context limits mid-task, forcing an ungraceful truncation instead of a planned compression.
Stuffing vs. Retrieval: The Core Trade-off
The core trade-off is that stuffing context upfront is simpler to build and feels safer, while retrieval on demand is more work to engineer but scales to tasks and knowledge bases that could never fit in a window at all. Neither is universally correct — the right choice depends on how much the relevant material varies step to step.
Stuffing wins when the material is small, stable, and needed on nearly every call — a style guide of a few hundred lines, a fixed schema, a short policy document. Retrieval wins when the material is large, variable, or only relevant to a subset of steps — a multi-thousand-file codebase, a customer history spanning years, a document corpus that changes weekly.
| Dimension | Stuffing (preload everything) | Retrieval on demand |
|---|---|---|
| Engineering effort | Low — concatenate and send | Higher — needs a retriever, ranking, chunking |
| Cost per call | High and roughly constant | Lower, scales with what's actually relevant |
| Freshness | Stale the moment source material changes | Naturally current if the retrieval source is |
| Scales to large corpora | No — hits window limits fast | Yes — the corpus can be arbitrarily large |
| Failure mode when wrong | Dilutes attention, buries key facts mid-window | Misses a fact the retriever ranked too low |
| Best fit | Small, stable, universally-needed material | Large, variable, or step-specific material |
Neither approach eliminates the need for judgment. Retrieval shifts the risk from "the window is too full" to "the retriever picked the wrong slice," which is a different failure mode, not a solved one. A retrieval system with poor ranking can silently omit the one paragraph that mattered, and unlike an overstuffed context, that failure often produces no visible warning at all.
Worked Example: Budgeting a Code-Review Agent
A code-review agent is a clean case study because its three candidate context sources — the diff, the style guide, and the file being changed — have wildly different sizes and stability, which is exactly what should drive the stuffing-versus-retrieval decision for each.
- The diff (task state, always include in full). It's small, it's the actual object of the review, and every line of it is relevant to this specific call. There's no version of this task where you'd retrieve only part of the diff — full inclusion is correct here.
- The style guide (retrieved facts, include only the relevant slice). A full engineering style guide can run to thousands of lines covering languages and patterns the current diff never touches. Retrieving only the sections relevant to the diff's language and the specific rules it's likely to trigger — naming conventions, error-handling patterns — keeps this tier lean without losing coverage where it counts.
- The file being changed (retrieved facts, include surrounding context, not the whole repo). The agent typically needs the function or class being modified and its immediate imports, not the entire file and certainly not the entire repository. Pulling in unrelated modules "in case they're relevant" is the textbook stuffing mistake — it adds tokens that compete for attention against the diff itself.
Why this ordering matters in practice
Put the diff and the most load-bearing style-guide rules near the edges of the context — start or end — rather than the middle, since that's where the "Lost in the Middle" degradation is smallest. If the style guide can't be reduced to a short slice, that's usually a sign the retrieval step ran too broad a query rather than a reason to just paste in more.
This example generalizes past code review. Any agent step has an equivalent of "the diff" (the thing actually being acted on), an equivalent of "the style guide" (a large reference corpus, mostly irrelevant per-call), and an equivalent of "the file" (immediately surrounding context, partially relevant). Naming those three roles explicitly for any given step is most of the budgeting work.
Making the Trade-off Visible, Not Implicit
Context budgeting fails silently more often than it fails loudly — an agent that's overloaded with irrelevant context usually just produces subtly worse output, not an error, which makes the problem hard to catch by watching for crashes. The fix is to make the decision explicit at design time rather than discover it in production behavior.
This also connects to tool access design more broadly. A step that has been granted access to a broad set of tools or data sources tends to accumulate context defensively, pulling in whatever those tools can return "just in case." Scoping tool access tightly, as covered in least-privilege agent tool access, constrains what a step can even attempt to stuff into its own context in the first place.
Context budgeting inside the larger agent architecture
Context budgeting doesn't happen in isolation — it's one piece of the broader discipline covered in the agentic workflows complete guide, alongside goal-setting, tool scoping, and evaluation. Treating it as a standalone checklist item, disconnected from how steps and tools are designed, tends to produce a budget that looks reasonable on paper but drifts back toward stuffing the moment a step gets complicated.
It's also worth borrowing from adjacent PM disciplines here: the same instinct that separates a real customer need from a nice-to-have feature request, as in jobs-to-be-done analysis, applies to context. Ask what job a piece of context is actually hired to do for this step, and cut it if the honest answer is "it might be useful."
Key Takeaways
- Rank context by priority, not by convenience: instructions, then task state, then retrieved facts, then history — spend the scarcest window space on the tiers that change least and matter most.
- More context is not automatically better context — the "Lost in the Middle" research shows models retrieve edge-of-window information more reliably than mid-window information, so placement matters as much as inclusion.
- Stuffing is simpler but doesn't scale; retrieval is more engineering work but is the only approach that works once the source material is larger than the window.
- A code-review agent's three inputs — diff, style guide, target file — map to a general pattern: the object being acted on, a large mostly-irrelevant reference corpus, and immediately surrounding context.
- Make the budget explicit at design time by listing required inputs per step; an oversized list is a visible signal that a step is asking for more than it needs.
- Context budgeting compounds with tool scoping and goal-writing — a step with broad tool access or a vague goal tends to pull in more context defensively, regardless of how disciplined the budgeting rules look on paper.
Frequently Asked Questions
How big should an agent's context window budget be?
There's no fixed number — it depends on the task, but a useful default is to keep instructions and task state under half the available window, reserving the rest for retrieved facts and history. The right size is whatever leaves enough headroom that a single step's retrieval doesn't force out task state the agent still needs.
What's the difference between context budgeting and prompt engineering?
Prompt engineering is about how you phrase instructions; context budgeting is about what content earns a place in the window at all, regardless of phrasing. They interact — a well-budgeted context still needs well-written instructions — but budgeting is fundamentally an allocation decision, not a wording decision.
Does a bigger context window make budgeting unnecessary?
No — larger windows reduce how often you hit a hard limit, but the "Lost in the Middle" findings suggest retrieval quality still degrades with distance from the window's edges even when there's technically room to spare. A bigger window makes stuffing less likely to fail loudly, which can make the subtler quality cost easier to miss, not less real.
Should I always retrieve on demand instead of stuffing context?
No — for small, stable material needed on nearly every call, like a short style guide or a fixed schema, stuffing is simpler and cheaper to build than a retrieval pipeline. Retrieval earns its complexity when the material is large, changes often, or is only relevant to a subset of steps.
How do I know if my agent's context is overstuffed?
A common signal is that output quality doesn't noticeably improve, or even degrades, as you add more context to a step — that's often a sign the added material is diluting attention rather than adding signal. Explicitly listing required inputs per step, rather than describing the step in prose, tends to surface an oversized request before it ever reaches the model.