Prompt Caching, Explained for PMs: Stop Re-Paying for Your System Prompt

Prompt caching lets an LLM provider store the tokens in your stable prompt prefix — a system prompt, tool definitions, retrieved context — so repeated calls re-read those tokens at a steep discount instead of re-billing them at full price. The catch: caching is a strict prefix match, so ordering and stability determine whether you save anything at all.

Quick answer: Put everything that never changes (system prompt, tool schemas, boilerplate instructions) first and mark it cacheable; put per-request content (the user's question, a timestamp, a session ID) last. Cache reads cost roughly a tenth of normal input pricing. On a 4k-token system prompt called 10,000 times a day, that ordering discipline is the difference between a five-figure and a four-figure monthly line item.

What prompt caching actually is

Prompt caching is a provider-side mechanism that stores the exact token sequence of a prompt's stable prefix so subsequent calls can skip re-processing it. Instead of paying full input price for your system prompt and tool definitions on every single call, you pay a small write premium once and a fraction of the price on every read after that, provided the prefix bytes match exactly.

Every major provider — Anthropic, OpenAI, Google — now ships some version of this, because the economics of production LLM apps demanded it. A PM building an AI feature typically ships a system prompt loaded with role instructions, formatting rules, few-shot examples, and increasingly a slab of retrieved context (RAG chunks, a knowledge base excerpt, a customer's account history). None of that changes between calls from different users asking different questions. Without caching, you re-pay for all of it, every single time, forever.

The core mechanic is a prefix match, not a semantic cache. The provider looks at the exact byte sequence up to a marked cutoff point and checks whether it matches a previously cached sequence. If one character anywhere in that prefix differs — a timestamp, a reordered JSON key, a different tool listed — the match fails and everything after that point in the prefix is billed at full price again. This is mechanically different from semantic caching for LLM responses, which caches whole answers for similar-meaning questions; prompt caching caches input tokens for byte-identical prefixes, and the two techniques solve adjacent but distinct cost problems.

Why this matters more as prompts grow

Teams shipping durable AI products tend to accumulate prompt weight over time: more edge-case instructions, more few-shot examples that fixed a bad output once, more retrieved context stitched in per call. A prompt that started at 500 tokens creeps to 4,000, 8,000, sometimes tens of thousands. Every one of those tokens gets billed as input on every call — unless caching intervenes.

This is exactly the kind of decision a PM should be running through a structured cost framework rather than intuition. Our complete guide to trade-off analysis covers the broader discipline of weighing cost, latency, and quality levers against each other before committing engineering time to any one fix — prompt caching is one of the cheapest levers in that toolkit, which is exactly why it's worth understanding before reaching for costlier ones.

How the discount and TTL actually work

Cached tokens are billed in two ways: a write the first time a prefix is stored (a small premium over normal input price, since the provider has to do extra work to persist it) and a read on every subsequent call that matches the same prefix, billed at roughly a tenth of standard input pricing. The cache entry expires after a time-to-live (TTL) window, typically five minutes of inactivity, with some providers offering a longer paid tier.

The TTL is the part PMs most often get wrong in their mental model. It is not "cached forever once written" — it's a rolling expiration that resets (on most providers) each time the cache is read. If your app gets a burst of ten calls in the same minute, the first call pays the write premium and the following nine read at the discount. If your traffic is bursty with long gaps — say, one call every twenty minutes — the cache cools down between bursts and you pay the write premium again on the first call of each burst, getting none of the discount's benefit for low-frequency paths.

Practically, this means prompt caching is a strong win for high-frequency, short-gap traffic — a chat product, a customer-support copilot, an internal tool used continuously through the workday — and a weaker or even negative win for sparse, bursty traffic where most calls arrive after the TTL has already lapsed. Before rolling this out, a PM should check actual call-frequency data for the feature in question, not just assume "we have a big prompt, so caching will help."

Extended TTL options

Some providers now offer an extended TTL (an hour instead of five minutes) at a higher write premium, aimed at exactly the bursty-traffic case above. It's a genuine lever, but it only pays off if your read volume within that hour is high enough to amortize the larger upfront write cost — worth modeling explicitly rather than flipping on by default.

The ordering rule: stable content first, variable content last

Caching only works because everything before a given point in the prompt is treated as one unbroken sequence — so the single most important implementation rule is: put content that never changes at the very front of the prompt, and content that changes on every call at the very end. Anything that varies mid-prefix breaks the match for everything downstream of it.

Concretely, a well-ordered prompt for a support-copilot use case looks roughly like this, top to bottom:

1. System instructions — role, tone, output format rules. Never changes across calls.
2. Tool/function definitions — the schema for any tools the model can call. Stable per deployment.
3. Static few-shot examples — if you use them, and they don't rotate per user.
4. Retrieved/RAG context relevant to this session — semi-stable within a session, but should still be placed after the fully-static block above it.
5. The user's actual message and any per-call metadata (timestamps, session IDs) — always last, always variable.

A common and costly mistake is interleaving a timestamp or a request ID into the system prompt itself — e.g. "Current time: 2026-07-10 14:32:01" injected at the top for the model's situational awareness. That single line invalidates the cache on every single call, because the prefix is never byte-identical twice. If the model genuinely needs the current time, put that line at the very end of the prompt, immediately before the user's message, so it doesn't sit inside the cached region.

The ordering rule generalizes past caching, too: this is the same principle behind batching for throughput vs. latency trade-offs — grouping stable, predictable work separately from variable, per-request work tends to unlock efficiency wherever LLM costs are structured around what's shared versus what's unique per call.

What breaks the cache silently

Several patterns invalidate a cache without throwing any error — the call just quietly reverts to full price, and a PM watching only total spend (not cache-hit rate specifically) may never notice:

• Non-deterministic serialization. If your backend serializes a JSON object with unsorted keys, or a tool list built from an unordered set, the byte sequence can differ between calls even though the logical content is identical.
• A conditionally-included system section. "If flag X, append paragraph Y" creates a distinct prefix per flag combination — each combination gets its own cache entry, fragmenting your hit rate.
• Per-user personalization baked into the system prompt. Injecting a user's name or account tier into the system-prompt text (rather than later, in the user turn) means no two users ever share a cached prefix.
• Retrieved context that reorders on every call even when the underlying documents haven't changed — a retrieval step that doesn't sort results deterministically.

The fix for all four is the same: audit what actually sits before your cache breakpoint, and move or stabilize anything that isn't guaranteed byte-identical across calls.

The math: a 4k-token system prompt at 10,000 calls a day

Here's a worked example showing why this is worth a PM's attention rather than an engineering afterthought. Assume a support-copilot feature with a 4,000-token system prompt (instructions, a handful of few-shot examples, and static tool definitions), called 10,000 times per day, using a mid-tier model priced around $3 per million input tokens.

Without caching, every one of those 10,000 calls pays full price for the entire 4,000-token prefix:

With caching, assuming traffic is frequent enough that most calls land within the TTL window (a reasonable assumption at 10,000 calls/day, which is roughly one call every 8-9 seconds on average), the vast majority of those calls read from cache instead of reprocessing:

That's roughly an 88% reduction on the prefix-token portion of the bill — call it comparable to what a properly-cached system prompt can realistically deliver at this volume, though your own multiplier depends on your provider's exact pricing and your actual cache-hit rate. The variable portion of each call (the user's own question, the model's output tokens) is unaffected by caching either way — this saving applies specifically to the stable prefix, which is exactly why prompts with large, unchanging systems sections benefit most and short, highly-variable prompts benefit least.

The number that matters for your own math is your cache-hit rate, not the theoretical maximum. If your traffic pattern is bursty with long gaps, run the same calculation with a higher write-to-read ratio before promising this saving to a stakeholder.

Where prompt caching sits versus other cost levers

Prompt caching is one of several cost levers a PM can pull, and it's usually not the first thing to reach for if the underlying problem is "we're using too expensive a model" rather than "we're re-billing the same tokens." Knowing which lever addresses which problem avoids wasted engineering cycles.

Prompt caching and model routing are not mutually exclusive — a well-optimized pipeline typically uses both: cache the stable prefix regardless of which model handles the request, and route the variable part to the cheapest model that can handle it. Reaching for a smaller or cheaper model before confirming your prefix is even cached is a common ordering mistake, since a smaller model still re-bills the same uncached prefix on every call.

A structured way to decide

Key Takeaways

• Prompt caching bills a stable prefix once and discounts repeated reads — roughly a tenth of standard input price on a cache hit, versus a ~1.25x premium on the initial write.
• Ordering is the whole implementation: stable content (system prompt, tool definitions) must come first; variable content (timestamps, user questions, session IDs) must come last, or the cache silently breaks.
• TTL windows mean caching favors frequent traffic. Bursty, sparse call patterns get less benefit because the cache cools down between bursts and re-pays the write premium.
• Silent invalidators are the most common failure mode — non-deterministic serialization, conditionally-included prompt sections, and per-user text baked into the system prompt all quietly break the match without any error message.
• On a 4k-token prompt at 10,000 calls/day, caching can plausibly cut prefix-token costs by roughly 85-90%, but the real number depends entirely on your actual cache-hit rate — measure it, don't assume it.
• Caching is usually the cheapest lever to pull before considering a smaller or cheaper model, since it has no quality trade-off at all — a framework like Prodinja's Trade-off Triangle can help sequence that decision correctly.