Packing the Window: Why the Order of Context Changes the Answer

Large language models don't treat every token in the context window equally: they recall and use information at the very start and the very end far more reliably than information buried in the middle. This is the lost-in-the-middle effect, and it means the order you pack context in is a design decision, not an afterthought.

Quick Answer: Models weight the start and end of a context window more heavily than the middle — a pattern researchers named "lost in the middle." Put your most important instructions and evidence at the edges, use supporting material as filler in between, and run a needle-in-a-haystack test to check whether your own prompts have a middle-neglect problem.

What the Lost-in-the-Middle Effect Actually Is

Lost in the middle is the finding that a model's ability to use a specific fact depends heavily on where that fact sits in the prompt, not just on its relevance. Accuracy traces a U-shape: high when the fact is near the start, high when it's near the end, and measurably lower for everything between — one of several forces covered in the complete guide to context engineering.

The clearest evidence comes from "Lost in the Middle: How Language Models Use Long Contexts," a 2023 study out of Stanford and UC Berkeley led by Nelson Liu and Percy Liang. The researchers ran multi-document question-answering and key-value retrieval tasks, moving the one relevant document to different positions in an otherwise-irrelevant context and re-measuring accuracy each time.

The pattern held across several model families and context lengths tested at the time. Performance was strongest with the answer at the very first or very last position, and dropped by a wide margin — often by tens of percentage points, in some configurations below what the same model scored with no relevant document in context at all — when the answer sat in the middle of a long document list.

That last detail is the uncomfortable part. Retrieval pipelines routinely assume that surfacing the right chunk is the hard part and feeding it to the model is a formality. The lost-in-the-middle result says feeding it in the wrong position can be worse than not retrieving it precisely enough in the first place.

The takeaway for anyone assembling prompts programmatically: where you place a fact is not a cosmetic choice. It sits closer to a routing decision, on par with deciding which fact to include at all.

Why Position Beats Relevance: The Mechanics Behind the Curve

Position outperforms relevance because of how autoregressive transformers process order and how they were trained, not because a model is being careless. Attention patterns and training-data conventions both create structural pressure toward the start and end of a sequence, leaving the middle comparatively under-attended regardless of content quality.

A few mechanisms compound:

1. Causal attention direction — decoder-only models process tokens left-to-right, so early tokens accumulate more paths of influence over everything that follows.
2. Training-data conventions — task-formatted training examples often place instructions and questions near the boundaries, teaching the model that boundary positions are where "the important part" lives.
3. Recency at generation time — tokens closest to where the model starts generating exert outsized influence, similar to how the most recent turns in a chat dominate short-term steering.

This isn't unique to machines. Cognitive psychologist Bennet Murdock documented the human version in 1962: the serial position effect, where people recall the first and last items of a list far better than the middle ones. The analogy isn't a claim that transformers "think" like people, but the resulting curve shape is strikingly similar — part of why the LLM finding was so immediately legible to researchers.

Greg Kamradt's "Needle in a Haystack" test made the effect visible at industry scale. By hiding a single sentence at varying depths inside long documents and asking models to retrieve it, Kamradt produced heatmaps that several labs now reference when discussing long-context reliability: dark bands in the middle, bright edges at top and bottom.

The size of the dip isn't uniform across model families, or even across versions from the same provider. Some show a steep drop past the halfway point; others degrade more gradually, and heavily instruction-tuned models sometimes show a smaller effect than their base counterparts. That variability is exactly why the template later in this article is a starting point, not a fixed rule — the test further down is what tells you whether your specific model and prompt shape actually need it.

Longer context windows don't fix this on their own. A bigger window just gives a model more room to lose things in the middle of — it doesn't change the shape of the curve.

An Ordering Template: Instructions, Key Evidence, Supporting Context, Query

A reliable default is to sequence a prompt in four zones — instructions, key evidence, supporting context, and the query — so the two positions a model attends to most, the start and the end, are never occupied by filler. Everything replaceable goes in the middle.

The four-zone template

Deciding what actually qualifies as "key evidence" is its own prioritization problem, not a formatting one. It's the same discipline you'd use to boil a stack of interviews down to the job the customer is actually trying to get done, or to find the one moment on a customer journey map that actually explains churn. Not everything you have is equally load-bearing — packing order just forces you to rank it honestly instead of dumping it all in and hoping the model sorts it out.

The long-document exception

For genuinely long reference material — a full contract, a large log file, several retrieved documents concatenated together — Anthropic's own long-context prompting guidance for Claude recommends a variation: put the bulk documents near the top of the prompt, ahead of the instructions, and keep the query at the very end. Anthropic's documentation reports this ordering can noticeably improve performance for Claude specifically, apparently because the model attends better to material sitting above the query than below it.

That refinement doesn't break the four-zone template so much as relabel zone one: when "supporting context" is actually a large document dump, promote it ahead of the instructions rather than burying it after them. The invariant that survives every variant is simpler than the template itself — never let the query or the decisive fact land in the middle third of the prompt.

This is a context engineering problem, not a prompt engineering one: the wording of an instruction rarely matters as much as the architecture around it. If that distinction feels fuzzy, the difference between context and prompt engineering is worth reading before going further, since ordering decisions live entirely on the context side of that line.

A Test to Detect Middle-Neglect

The most direct test is a scaled-down version of the needle-in-a-haystack method: insert a known fact at several positions in a realistic-length version of your actual prompt, ask the same question each time, and score whether the answer reflects the fact. A dip in the middle positions confirms middle-neglect.

1. Pick a realistic context length. Use the size your production prompts actually reach, not a toy example — the effect is negligible at a few hundred tokens and can be severe past several thousand.
2. Write a fact the model can only get right by using it. A specific number, name, or decision that isn't guessable from general knowledge or the rest of the context.
3. Insert it at five positions. Roughly 0%, 25%, 50%, 75%, and 100% depth through the packed context.
4. Ask the same question at each position, several times. Single runs are noisy; a handful of repetitions per position gives a usable signal.
5. Score accuracy@position and look at the shape. A flat line means packing order isn't your bottleneck. A dip in the middle confirms it is, and tells you which zone to fix first.

Run this before switching models, not only before shipping a feature. Different model families sit at different points on the U-curve, and a template tuned for one can quietly regress on another.

Packing Order in Practice: RAG, Retrieval, and Multi-Turn Threads

In production systems the ordering decision usually shows up in two places: how retrieved chunks get sequenced in a RAG pipeline, and how conversation history gets carried across turns. Both need the same fix — stop defaulting to whatever order retrieval or chat history happened to produce.

In retrieval pipelines, the default is almost always "sort by similarity score, concatenate top-k in that order." That puts your single most relevant chunk wherever it happens to rank, which is frequently the middle of the packed list once you've pulled in five or ten chunks. Re-rank after retrieval so the top-scoring chunk sits at an edge, not buried inside a flattened list.

Which chunks even make it into that ranked list depends on retrieval quality upstream. Choosing an embedding model well is what determines whether the "key evidence" zone has the right candidate in it at all, before ordering ever becomes relevant.

In multi-turn conversations, older turns drift toward the middle of the window as the thread grows, exactly where attention is weakest. Two patterns help:

• Summarize older turns into a compact block placed near the instructions, so it benefits from primacy without the full original token cost.
• Re-state the live task at the end of every turn, rather than relying on the model to recall it from several exchanges back.

The caching trade-off

Reordering content on every request has a cost most teams don't budget for. Prompt-caching features, including Anthropic's, reward a stable, unchanged prefix — the discount only applies when the same tokens appear at the start of consecutive requests. Re-ranking retrieved chunks or repositioning key evidence on every call can quietly break that stability, trading a cache-hit discount for an accuracy gain.

The practical compromise is to cache the parts of the packet that genuinely don't change per request — system instructions, tool definitions, static background — and reserve reordering for the smaller, request-specific slice: the key evidence and the query. You rarely need to reshuffle an entire multi-thousand-token packet to fix a middle-neglect problem; usually moving one or two chunks is enough.

Where Prodinja Fits

Prodinja doesn't auto-solve packing order, but its context packet is built around exactly this problem. It separates system instructions, retrieved content, and the token budget into distinct pieces rather than one flattened prompt string, which gives you a structure to reason about what goes where in the window.

That separation matters because you can't fix an ordering problem you can't see. If instructions, evidence, and retrieved filler are concatenated into one opaque block before it reaches the model, there's no lever to pull when a needle-in-a-haystack test comes back with a dip.

The anatomy of a context packet covers how those pieces map to a request in more detail. Treat it as scaffolding for the decisions in this article, not a substitute for making them — you still decide what counts as key evidence and where the budget gets spent.

Key Takeaways

• Position predicts usage as much as relevance does — a model can fail to use a fact it was given, purely because of where that fact sat in the prompt.
• The lost-in-the-middle curve is U-shaped: strong at the start, strong at the end, weakest in the middle third, and the dip can be severe enough to underperform having no relevant content at all.
• A four-zone template — instructions, key evidence, supporting context, query — is a reliable default, with a documented exception for very long reference documents, which some model guidance recommends placing ahead of instructions.
• Bigger context windows don't fix this — they just give a model a longer middle to lose things in.
• Test for middle-neglect directly with a needle-in-a-haystack style check across five or so positions before trusting a template in production.
• RAG re-ranking and multi-turn summarization are packing-order problems in disguise — both need the same fix of deliberately promoting decisive content to the edges.
• Deciding what counts as "key evidence" is a prioritization exercise, not a formatting one — the same discipline used to distill customer interviews or a journey map down to what actually matters.