Context Engineering: Deciding What Belongs in the Prompt vs Retrieved at Runtime

Context engineering means splitting your AI feature into two layers: a stable instruction layer (persona, rules, output contract) that rarely changes, and dynamic runtime context (user data, retrieved documents, session state) assembled fresh on every call. The heuristic is simple — if it changes per request or per user, it doesn't belong in the system prompt.

Quick Answer: Put anything that is true for every call — role, constraints, output format — in the system prompt. Put anything that varies by user, session, or document in retrieved/injected context. If you're re-writing the system prompt per request, you've built retrieval by accident.

Why "Just Put It All in the System Prompt" Fails

Teams default to stuffing everything into the system prompt because it's the first text box they see, and it feels safest — nothing gets forgotten. The direct 40-60 word answer: this approach fails because system prompts get re-sent on every single call, so anything volatile inside them multiplies your token cost, stales silently, and makes the model's actual instructions harder to find.

Three concrete failure modes show up in production RAG and context-heavy features:

1. Cost multiplies invisibly. A 4,000-token system prompt with embedded user profile data gets billed on every turn of every conversation, not once per session.
2. Staleness compounds. User data baked into a prompt template at request-build time can drift from the source of truth within minutes — inventory counts, ticket status, account tier.
3. Instruction-following degrades. Anthropic's own prompt engineering guidance and OpenAI's system-prompt documentation both note that models attend less reliably to instructions buried inside long, data-heavy blocks — the classic "lost in the middle" effect described in Liu et al.'s 2023 Stanford/UC Berkeley research on long-context retrieval.

The result is a prompt that's simultaneously expensive, wrong, and less obeyed. That's the "lazy" failure mode this article is named for — not laziness of effort, but laziness of architecture. Our companion piece on the system prompt as the new PRD covers why that instruction layer deserves the same rigor as a product spec; this article is about what should — and shouldn't — live inside it.

The Two-Layer Mental Model

The direct answer: every AI feature has exactly two kinds of context — the durable layer (who the model is, what it must never do, how it must format output) and the assembled layer (what it needs to know right now to answer this specific request). Confusing the two is the root cause of most context bloat.

The Durable Instruction Layer

This layer answers "what kind of assistant is this, always?" It should be:

• Persona and scope — role, tone, domain boundaries ("You are a support triage assistant for a B2B SaaS product; you do not give legal or medical advice").
• Non-negotiable rules — safety constraints, escalation triggers, things the model must refuse.
• Output contract — schema, format, length constraints. See our guide on structured outputs and shippable JSON for how to make this contract machine-enforceable rather than aspirational.
• Reasoning process — how to approach the task (chain-of-thought scaffolding, tool-use policy, when to ask clarifying questions).

None of this should reference a specific user, ticket, document, or session. If you find yourself writing "the user's name is {{name}}" inside this block, that's retrieval content wearing a system-prompt costume.

The Dynamic Runtime Context

This layer answers "what do I need to know to handle this request?" It's assembled per call, typically from:

• Retrieved documents (RAG chunks, knowledge base articles, prior tickets)
• User/session state (profile, permissions, current object being viewed)
• Conversation history (summarized or windowed)
• Tool outputs from earlier in the same turn

This layer is disposable and rebuilt every time. It should never be hand-edited into a prompt template — it should come from a retrieval or state-fetch step that runs before the call.

The Placement Heuristic: Two Questions

Here's the direct answer to "how do I decide where something goes": ask does it change per request? and does it change per user? If yes to either, it's runtime context. If no to both — it's true for every user, every time — it belongs in the durable instruction layer.

A useful second-order test: would this line, if wrong, embarrass the product or embarrass the user? Wrong instructions (a broken output contract) embarrass the product across every user. Wrong data (a stale ticket count) embarrass — or just confuse — one user, in one moment. Product-wide embarrassment risk belongs in the layer you review and version like code; user-specific risk belongs in the layer your retrieval pipeline is responsible for keeping fresh.

Edge Cases That Trip People Up

Some content genuinely sits in a gray zone. A few resolutions that hold up in practice:

• Feature flags / tenant configuration — treat as retrieved context, even though it changes slowly, because it changes per tenant, not per instruction version.
• Few-shot examples — these belong in the durable layer if they demonstrate the output contract generically; move them to runtime context if they're pulled from the specific user's own history (that's retrieval, not instruction).
• Long, static reference material (a full product manual) — even though it never changes per user, it should usually be retrieved on demand (via RAG) rather than pinned in the system prompt, because it's large and mostly irrelevant to any single request. Static and small stays in the prompt; static and large gets retrieved.

A Diagram of the Layers

Rendered as a simple stack, the flow from durable instruction to final assembled prompt looks like this:

┌─────────────────────────────────────────────┐
│  LAYER 1 — Durable Instruction (versioned)   │
│  • Persona / role / scope                    │
│  • Non-negotiable rules & refusals           │
│  • Output contract (schema/format)           │
│  • Reasoning / tool-use policy               │
│  → changes rarely; reviewed like code        │
└─────────────────────────────────────────────┘
                     +
┌─────────────────────────────────────────────┐
│  LAYER 2 — Retrieved / Assembled Context     │
│  • RAG chunks relevant to this query         │
│  • User profile / permissions / session      │
│  • Conversation history (windowed/summarized)│
│  • Tool outputs from earlier in the turn     │
│  → rebuilt fresh on every call               │
└─────────────────────────────────────────────┘
                     ↓
┌─────────────────────────────────────────────┐
│  FINAL PROMPT SENT TO MODEL                  │
└─────────────────────────────────────────────┘

The point of drawing it this way: Layer 1 is a release artifact — it has a version number, a changelog, and a review process, the same way our piece on versioning prompts like code argues. Layer 2 is a runtime computation — it has no version number because it's different on every call by design. Treating both layers with the same process (or worse, merging them into one blob) is where teams lose track of what actually changed between a good answer and a bad one.

A Worked Example: Moving User Data Out of the System Prompt

Consider a support-triage assistant where the original implementation built a system prompt per request that interpolated the current user's name, tier, open ticket count, and last five ticket subjects directly into the instruction block, alongside the persona and output rules. Every call re-sent that entire block.

The direct answer for what breaks: this pattern makes the system prompt effectively infinite in variety (one per user-state combination), which means it can never be cached, never be reviewed as a fixed artifact, and never be tested against a stable golden set — because the "instructions" are different every time.

Restructuring the same feature into two layers looks like this:

The cost effect is direct: with the instruction layer isolated and stable, it becomes eligible for prompt caching, which providers price at a fraction of full input-token rates — a well-documented mechanic in both Anthropic's and OpenAI's caching docs, though the exact discount varies by provider and content size. The confusion effect is equally direct: engineers debugging a bad response could finally ask "was this an instruction problem or a data problem?" because the two were no longer the same string.

None of this requires exotic tooling — it requires discipline about which string gets built once (and reviewed, versioned, tested) versus which gets built fresh every time. That discipline is exactly what's missing when teams treat prompt engineering as one undifferentiated skill; our complete guide to prompt design covers the broader craft this fits inside.

Where This Connects to Retrieval Quality, Not Just Prompt Hygiene

Splitting the layers isn't just a cost and readability exercise — it directly affects retrieval quality, because a bloated system prompt competes for the model's attention budget against the retrieved chunks that actually answer the question. The direct answer: the more static filler occupies the context window, the less signal-to-noise your retrieved documents carry relative to total input.

This matters most in RAG-heavy features where retrieval precision is already imperfect:

• Retrieval-augmented generation surveys (e.g., Lewis et al.'s original RAG paper and subsequent retrieval-quality research) consistently find that irrelevant or redundant context in the window degrades grounded-answer accuracy, not just cost.
• A durable instruction layer that's lean leaves more effective context budget for the retrieved documents that are actually per-request signal.
• Teams that conflate "what the assistant always knows how to do" with "what the assistant needs to know right now" tend to over-retrieve and over-instruct simultaneously, compounding the noise problem.

This is also where context engineering intersects with product discovery work upstream — a feature that retrieves the wrong documents because nobody defined the underlying user need clearly enough is a discovery gap, not a prompt gap. If retrieval keeps missing what users actually need answered, it's worth revisiting the job the feature is meant to do, the way our JTBD guide and customer journey guide frame it — because no amount of prompt-layer discipline fixes a retrieval index built around the wrong questions.

How Prodinja Approaches This Split

Prodinja's Context Engineering tool is built around this exact durable-versus-assembled distinction, treating the instruction layer as a separate, reviewable artifact from the context assembled per request — rather than one prompt string that grows a little with every edge case. It's designed to walk a technical PM through classifying prompt content against the same "changes per request / changes per user" questions this article uses, so the split is a deliberate design decision rather than something a team backs into after a cost spike.

That framing sits alongside Prodinja's other context-and-quality tools — a critique layer for spotting when eval coverage is thin, and a separate lens for reasoning through agentic workflow structure — all part of the Studio's broader approach to treating prompt and context decisions as product artifacts, not throwaway strings.

Key Takeaways

• Ask two questions for every piece of prompt content: does it change per request, and does it change per user? Either "yes" means it belongs in retrieved context, not the system prompt.
• The durable instruction layer (persona, rules, output contract) should be small, versioned, and reviewed like a spec — not rebuilt per call.
• Large-but-static reference material still belongs in retrieval, not the prompt, even though it doesn't vary per user — size and relevance-per-request matter as much as volatility.
• A bloated system prompt hurts more than cost — it competes with retrieved documents for the model's attention and degrades grounded-answer quality.
• Isolating the instruction layer unlocks prompt caching, which materially changes the cost structure of high-volume AI features.
• Context engineering is a retrieval-quality problem as much as a prompt-hygiene problem — noisy instructions and noisy retrieval compound each other.