Prompt Caching: The Cost Lever Hiding in Your Repeated Context

Prompt caching lets you pay full price once for a repeated chunk of context — a system prompt, a style guide, a product doc — and pay roughly a tenth of that price on every subsequent call that reuses it. Most teams never turn it on because it requires restructuring how the prompt is assembled, not just flipping a flag. For a high-volume feature with a big fixed prompt, that's often the single largest lever on your AI cost line.

Quick Answer: Prompt caching stores a processed prefix of your prompt (system instructions, docs, few-shot examples) so repeat calls skip reprocessing it. Cached reads cost roughly 10% of normal input pricing. The catch: it only works if the cached portion is byte-identical across calls and placed first, before anything that changes.

What Prompt Caching Actually Does

Prompt caching is a mechanism where an LLM provider remembers the "compiled" state of a prompt prefix so it doesn't have to reprocess those tokens from scratch on the next call. Instead of billing every input token at full price, repeated content is billed at a steep discount — often around 90% off — as long as it matches exactly.

Think of it like a spreadsheet with a frozen header row. Every time you scroll (send a new request), the header doesn't need re-rendering — it's already there. The "body" of the sheet (your changing user question) still gets processed fresh. Caching works the same way: stable content first, volatile content last.

This matters because a huge share of real-world LLM cost is not the user's question — it's the scaffolding you resend every single time: a lengthy system prompt describing tone and rules, a product knowledge base, a style guide, a set of tool definitions, or a handful of few-shot examples. If that scaffolding is 3,000 tokens and the user's actual question is 50 tokens, you've been paying full price for the 3,000 tokens on every call, forever.

Why This Isn't Automatic

Caching is a prefix match, not a semantic match. If a single character changes anywhere in the cached portion — a timestamp inserted into the system prompt, a reordered JSON key, a randomly-generated request ID placed near the top — everything after that point in the prompt is invalidated and reprocessed at full price. This is the part that trips up most implementations: the feature works exactly as documented, but a well-intentioned datetime.now() in the system prompt silently defeats it on every single call.

For background on how LLMs bill in the first place, see tokens, not words: the AI billing unit — caching is a discount applied to that same token-based meter, not a separate pricing system.

Eligibility: What Can (and Can't) Be Cached

Anything that sits in the request's tools, system, or messages fields can be cached — but only content above a minimum length, and only when it renders in the same byte order every time. Below that threshold, caching silently does nothing: no error, just a cache-write cost with zero benefit.

Practically, the render order for most providers is fixed: tools → system → messages. A cache marker placed at the end of your system prompt caches both the tool definitions and the system prompt together, which is usually the highest-value single breakpoint in a typical application.

Minimum prefix length varies by model, typically in the range of roughly 1,000 to 4,000 tokens depending on the model tier. A short system prompt under that floor simply won't cache — which is one reason caching often goes unnoticed as "not working" when really it was never eligible.

TTL and the Economics of Cache Writes

A cache entry isn't permanent — it expires after a time-to-live (TTL) window, commonly a short window like five minutes, with a longer option (around an hour) available at a higher write cost. Choosing the wrong TTL is the second most common way teams undermine their own savings.

Here's the tradeoff: a cache write costs more than a normal input token — often on the order of 25% more for the short TTL, or roughly double for the long TTL. A cache read costs a fraction of normal price. That means caching only pays off if you get enough reads before the entry expires.

1. Short TTL (minutes): Break-even after about two requests within the window — cheap to test, works well for bursty, back-to-back traffic (a chat session, a batch job).
2. Long TTL (roughly an hour): Needs about three or more requests to break even, because the write premium is higher — better for a feature with steady but spaced-out traffic, like a support tool used throughout a business day.
3. No caching: If your feature gets one call every few hours with a unique context each time, don't bother — you'll only ever pay the write premium and never collect the discount.

The practical rule: if your traffic pattern already refreshes the cache faster than it expires, you don't need to manually "re-warm" it — real traffic keeps it alive on its own.

Design Patterns: Put Stable Content First

The core design discipline is ordering your prompt by volatility, from most stable to most volatile, and placing the cache marker at the boundary. Get the ordering right and most of the savings happen automatically; get it wrong and no amount of configuration fixes it.

The Four-Layer Prompt

A useful mental model for structuring any LLM-backed feature's prompt:

• Layer 1 — Frozen system instructions. Tone, role, output format rules. Changes maybe once a quarter. Goes first, always.
• Layer 2 — Static reference material. Product docs, policy text, a style guide, tool schemas. Changes rarely. Goes right after Layer 1.
• Layer 3 — Session-scoped context. Prior conversation turns, a user's profile facts loaded once per session. Changes per-session, not per-call.
• Layer 4 — The live turn. The user's actual message, retrieved-just-now search results, the current timestamp. Changes every single call — this goes last, after the cache boundary, and is never part of the cached prefix.

The failure mode to watch for: an engineer adds "today's date is X" to the top of the system prompt so the model has date awareness. That one line, sitting in Layer 1's position but behaving like Layer 4, invalidates the entire cache on every call. The fix is simple once you see it — move dynamic facts to the end of the message list, not the top of the system prompt.

Common Anti-Patterns

This is a close cousin of the reproducibility problems covered in why LLM outputs aren't fully reproducible — small, invisible differences in what you send produce outsized, invisible differences in what happens next, whether that's a changed cache hit or a changed output.

A Savings Estimate for a High-Volume Feature

Consider a support-triage feature: a 3,000-token system prompt (policies, tone rules, a few dozen example classifications) plus a 100-token incoming ticket, run 50,000 times a month. Without caching, every call pays full input price for the whole 3,100 tokens.

With caching, only the first call in each cache window pays the write premium for the 3,000-token prefix; every other call within that window pays the steep cached-read rate for those 3,000 tokens, plus full price for the 100-token ticket. Directionally, if roughly 90% of the traffic lands as cache reads rather than fresh writes, the system-prompt portion of the bill can shrink by somewhere in the neighborhood of 80-90%, because reads are priced at a fraction of a fresh input token and only the small, genuinely variable part of the prompt keeps paying full price.

The size of the win scales with two things: how large the fixed portion is relative to the variable portion, and how tightly packed your call volume is inside the TTL window. A feature with a tiny system prompt and a huge unique document per call won't see much benefit — there's nothing stable to cache. A feature like the support-triage example, dominated by a large fixed prompt and a small variable payload, is close to the ideal case.

This is a directional estimate, not a guarantee — actual savings depend on your specific prompt sizes, call volume, and TTL choice. Model your own numbers with your provider's published cache pricing before committing to an architecture change.

How Prodinja's Context Engineering Practice Fits In

Knowing that caching exists is different from knowing which parts of your specific feature's context are actually stable enough to cache — that requires looking honestly at what changes call-to-call versus what's copy-pasted every time. Prodinja's Context Engineering practice is designed to walk a PM through exactly that: laying out a feature's context inputs and surfacing which pieces are frozen scaffolding versus which are genuinely live, so a caching-friendly prompt structure is a natural output of the exercise rather than an afterthought bolted on by an engineer under deadline pressure.

This sits alongside Prodinja's broader Studio tools — the same instinct that asks "what's stable versus volatile in your context" is the instinct behind good prompt design generally, which is why the practice is framed as a reasoning tool for PMs, not a black box that hands back a verdict.

Key Takeaways

• Prompt caching discounts repeated prefixes, typically to around a tenth of normal input pricing, but only when the cached portion is byte-identical across calls.
• Ordering matters more than configuration: stable content (system prompt, docs, tool definitions) must come first; volatile content (the live question, timestamps, IDs) must come last.
• A single stray timestamp or UUID near the top of your prompt silently defeats caching — no error is thrown, the cache read count just quietly stays at zero.
• TTL choice should match your traffic pattern — short TTL for bursty back-to-back calls, longer TTL for steady-but-spaced traffic, and skip caching entirely for rare, unique-context calls.
• The biggest wins come from high-volume features with a large fixed prompt and a small variable payload — support triage, classification, and structured extraction are classic candidates.
• Verify it's actually working by checking your provider's usage metrics for cache-read token counts — don't assume caching is active just because you added a marker.