Semantic Caching for LLM Features: Serve Answers You Already Paid For

Semantic caching stores an LLM's past responses indexed by what a prompt means, not its exact text, so a differently-worded question that asks the same thing returns instantly from cache instead of hitting the model. Done carefully, it cuts cost and latency together — done carelessly, an overbroad similarity threshold serves a wrong answer with total confidence.

Quick answer: Semantic caching matches new prompts against previously answered ones using embedding similarity, then returns the stored response once similarity clears a threshold. It works best on repetitive, non-personalized, non-time-sensitive queries, and fails silently — not loudly — when the threshold is set too loose.

If you're running a high-volume LLM feature — a support bot, an internal Q&A tool, a documentation assistant — a large share of what comes in is some version of a question you've already answered. Semantic caching is the lever that turns that repetition into savings. It's also one of the easier ways to accidentally ship a confidently wrong answer, which is why the mechanics matter more than the marketing pitch.

Exact-Match Caching vs. Semantic Caching: What's Actually Different

Exact-match caching returns a stored response only when the new request is byte-for-byte identical (or identical after simple normalization) to a previous one. Semantic caching instead compares the meaning of the new prompt to stored prompts using vector embeddings — the same representation underlying retrieval and search, walked through in our guide to LLM fundamentals — returning a match when similarity clears a set threshold, even if not a single word overlaps.

The two solve different problems and most production systems eventually run both, layered.

Anthropic's own documentation on prompt caching — an exact-match mechanism that reuses the computed state of a shared prefix like a system prompt or a long document — reports up to roughly 90% cost reduction and up to roughly 85% latency reduction on cache hits for long-context prompts. That's the ceiling for exact-match: real, meaningful, but only available when the same context recurs verbatim.

Semantic caching goes further because it doesn't require verbatim recurrence. The open-source project GPTCache, built by the team behind the Zilliz/Milvus vector database, is one of the most widely adopted implementations of exactly this pattern: embed the incoming prompt, search a vector index for the nearest stored prompt, and return its cached response if the distance clears a configurable threshold. That flexibility is also where the risk lives, which is the subject of the rest of this piece.

It's worth separating this from a related but distinct idea covered in our guide to routing requests across cheap, fast, and default models: caching avoids the model call entirely for a subset of traffic, while routing changes which model handles the calls that still need one. They're complementary levers, not substitutes, and the order you reach for them matters — more on that below.

The Cache-Hit-Rate Model: What You're Actually Saving

Your realized savings from semantic caching equal your duplicate query share multiplied by your threshold recall — the fraction of true duplicates your chosen similarity cutoff actually catches — multiplied by the cost or latency of the LLM call you're skipping. Most teams skip straight to picking a threshold without modeling this, then can't explain why the cache "isn't helping."

Break it into three inputs you can actually measure:

1. Duplicate query share (D) — the percentage of incoming queries that are near-duplicates of something already answered. Find this by clustering your query logs, not by guessing; this is genuinely discovery work, the same muscle described in our guide to product discovery applied to your own usage data instead of user interviews.
2. Threshold recall (r) — of the true duplicates in (D), what fraction does your similarity cutoff actually match? A stricter threshold lowers this number but also lowers your false-positive rate.
3. Effective hit rate (H) — approximately D × r, the share of total traffic actually served from cache.

To make this concrete, walk through a hypothetical (the shape of the math matters more than these specific numbers — pull your own from your logs):

Gross savings on skipped LLM calls: 56,000 × $0.006 ≈ $336/day. Subtract lookup overhead across all 200,000 requests (roughly $20/day), and net savings land around $316/day — order of $115K a year, at this volume and cost profile. The latency story is similarly stark: if a typical LLM call runs 1,000–1,500ms at p50 and a cache lookup runs 50–80ms, the 28% of requests served from cache drop by roughly an order of magnitude in response time.

That's the appeal in one sentence: semantic caching is one of the rare levers that improves cost and latency at the same time, instead of trading one for the other. Most optimization work — smaller models, shorter prompts, batching — improves one at the expense of, or independent of, the other.

Where Semantic Caching Fails: Personalization, Recency, and Confidently Wrong Answers

Semantic caching fails when the question is stable but the correct answer isn't — because the answer depends on who's asking, when they're asking, or on entities the embedding model doesn't weight heavily enough to distinguish. In all three cases, the failure is silent: the cached response reads exactly as fluent and confident as a fresh one.

Three recurring failure patterns are worth naming explicitly:

• Personalized answers. "What's my current plan?" or "What's the status of my last order?" are template-identical across every user but factually different per user. If the cache key doesn't include the user or account context, you'll serve one customer's data to another — a correctness failure, not just an annoyance.
• Time-sensitive answers. "Is the checkout page having issues right now?" or "What's the current promotion?" have stable phrasing and unstable truth. A cache with no TTL (time-to-live) will happily serve yesterday's correct answer as today's fact.
• Near-miss entity swaps. "What's the return policy for orders under $50?" and "...over $50?" differ by one word and can sit uncomfortably close together in embedding space, while requiring opposite answers. This is the sharpest version of the correctness risk: two questions that are almost duplicates but not quite.

A wrong answer from a fresh LLM call at least looks uncertain if you design for it. A wrong answer from cache looks identical to a right one — same tone, same formatting, same confidence. That gap is exactly the kind of failure mode explored in our piece on designing for LLM failure states: the UX problem isn't the error, it's the absence of any signal that an error occurred.

Multi-turn conversations add a fourth wrinkle. The same follow-up phrase ("what about next month?") means something entirely different depending on what was asked three turns earlier, and a cache that embeds the isolated turn — rather than the turn plus relevant context — will happily match it to the wrong prior thread. This is a live concern anywhere caching meets stateful, chained reasoning, which is exactly the territory covered in our guide to agentic workflows: the more steps and memory a system carries, the more careful the cache key has to be about which context it's actually matching on.

The practical response isn't to abandon semantic caching for these categories — it's to exclude them from caching candidacy or parameterize the volatile part out of the cache key (cache the policy lookup, not the user-specific account check; cache with a short TTL, not indefinitely).

Setting the Similarity Threshold: A Conservative Rule

The single highest-leverage decision in a semantic cache is the similarity cutoff, and the safe rule is: start narrow, and only widen the threshold when you've measured precision on adversarial near-misses, not just recall on obvious duplicates. Most teams do the opposite — they tune for hit rate first and discover the false-positive problem in production.

OpenAI's own embeddings guidance notes that cosine similarity scores for related-but-distinct sentences compress into a narrow, high band — which is precisely why a threshold tuned for general semantic search relevance is usually too loose for answer-equivalence caching, where "close enough" and "correct" are not the same thing.

A concrete way to reason about the trade-off (illustrative pattern, not a measured benchmark — validate against your own data):

Turn that into an operating procedure:

1. Set the initial threshold high — near the "literal rephrasing only" range — and accept a modest hit rate to start.
2. Build a labeled adversarial test set: pairs of queries that differ in one critical entity (a date, a number, a name, a negation) and should never match, alongside true duplicate pairs that should.
3. Measure precision on the adversarial set before you measure recall. A single wrong match on a negation pair is a blocking bug, not statistical noise — it's silent and it looks correct.
4. Only lower the threshold when precision on that adversarial set holds, and lower it in small increments, re-testing each time.
5. Exclude or parameterize volatile entities — dates, prices, quantities, account identifiers — from the embedded text, or exclude those query types from caching altogether.
6. Add a structural guardrail on top of similarity, not instead of it: a lightweight entity-diff check between the new query and the matched cache entry catches the cases embeddings miss.
7. Re-validate the threshold whenever the embedding model changes. A cutoff tuned for one embedding model's geometry does not transfer cleanly to another.

The underlying principle: treat every avoided LLM call as a small win, and every wrong cache hit as an incident. The asymmetry is real — one is a cost saving, the other is a trust problem — and your threshold should reflect that asymmetry, not split the difference.

Architecture: Where Caching Sits in the Pipeline

A production semantic cache is a filter that sits between your application and the model call, checked in order of cheapest and safest first: normalize and hash the prompt, check an exact-match cache, and only if that misses, embed the query and check a vector-similarity cache before ever reaching the LLM. This layering matters because exact-match is nearly risk-free and catches a meaningful share of traffic on its own.

The typical flow looks like this:

1. Normalize the incoming prompt (whitespace, casing, common boilerplate).
2. Check the exact-match cache (a hash lookup) — cheapest possible check, zero correctness risk.
3. On a miss, embed the prompt using a consistent embedding model (e.g., text-embedding-3-small-class models) and query a vector index — pgvector, RedisVL's semantic cache, or a dedicated vector store like Milvus.
4. Apply the threshold and any structural guardrails from the section above.
5. On a hit, return the cached response and log it distinctly as a cache hit for monitoring.
6. On a miss, call the LLM, then write the new prompt embedding and response into the cache with an appropriate TTL.

Redis's own semantic caching reference architecture (RedisVL) documents this same shape — a vector index in front of the model call with a tunable distance threshold — as a standard building block for cutting redundant calls in production RAG and chat systems. At the infrastructure layer, researchers behind the vLLM inference engine have documented a related idea one level down the stack: reusing key-value cache memory across requests that share a token prefix (PagedAttention) to cut serving cost and latency without touching the model itself. Same principle, different altitude — one operates on tokens, the other on meaning.

Monitoring matters as much as the initial threshold choice. Track hit rate, cost per resolved query, p50/p95 latency by cache-hit vs. cache-miss path, and — critically — a channel for flagging wrong cache hits, since these won't show up as errors on their own.

When Cost and Latency Both Flash Red

This is the scenario worth naming directly: a feature where both cost and latency are the binding constraints, not quality. If you're working through a structured trade-off analysis and cost and latency come out co-dominant rather than one clearly outranking the other, that's the specific signature caching addresses — because it's one of the few levers that moves both at once, rather than forcing you to trade one for the other the way model choice usually does.

That's the logic behind how Prodinja's Trade-off Triangle workspace is designed to respond to that pattern: when cost and latency flag as co-dominant, its recommendation steers toward standing up a caching layer before you touch model selection. Caching captures savings on repeated traffic regardless of which model runs underneath it.

Model routing — covered in the guide on routing across cheap, fast, and default models — remains the right lever for traffic that isn't a cache hit. The two aren't in competition; they apply to different slices of your volume.

Key Takeaways

• Semantic caching matches on meaning, not text — it uses embedding similarity to serve a stored answer to a differently-phrased but equivalent question, unlike exact-match caching, which only fires on literal duplicates.
• Model your hit rate before you build: effective hit rate is roughly duplicate query share multiplied by threshold recall — measure both from your own logs rather than assuming a number.
• Caching is one of the few levers that cuts cost and latency simultaneously, which is exactly why it belongs in the conversation whenever both show up as binding constraints together.
• Personalized, time-sensitive, and multi-turn-context-dependent queries are the classic failure modes — exclude them from caching, parameterize the volatile parts out of the cache key, or apply a short TTL.
• Set the similarity threshold conservatively and prove precision on adversarial near-misses before chasing recall — a wrong cache hit is a silent, confident failure, not a rounding error.
• Layer exact-match in front of semantic caching; it's cheaper, carries essentially no correctness risk, and catches a real share of traffic on its own.
• Caching and model routing are complementary, not competing — caching removes calls entirely for repeated traffic; routing optimizes which model handles what's left.