Prompt caching in production: exact vs semantic, and the break-even math nobody does

"How do we cache prompts with LLMs?" is a real technical-interview question at big tech now. The textbook answer is two techniques and a shrug about tradeoffs. The production answer is a piece of arithmetic almost nobody actually runs — and it's the difference between caching saving you money and quietly costing you more.

The two kinds of caching

Exact caching keys on a byte-identical prompt (usually the prefix). Same bytes → serve the cached result. Fast, cheap, and dumb: change one character at the front and the cache misses. On a metered API this is the "prompt cache" — reading cached input costs roughly a tenth of a fresh input token.

Semantic caching keys on meaning: embed the query, and if a past query is similar enough, reuse its answer. It catches paraphrases exact caching misses — but it can also return a confidently wrong answer when "similar enough" isn't actually the same question. You're trading a cheap exact check for an embedding lookup plus a similarity threshold you have to tune.

The tradeoff people skip: caching can lose money

Here's the arithmetic. Reading from cache is cheap (~0.1× a fresh input token), but writing to cache is dearer than a normal token (~1.25×). So caching a block is only worth it if it's read back enough times.

Over K uses of the same block:

no cache : K × 1.0
cache    : 1.25  +  (K − 1) × 0.1

Cache wins when K > ~1.28. In plain terms: used twice or more, caching wins; used once, caching is strictly worse than not caching at all. Every "cache everything" default ships some blocks that are written once and never read — pure loss. You can count those directly and stop doing them.

And a trap specific to local models

I run inference on a local vLLM lane, and there the billing field cache_read_input_tokens reads 0 on every request — which looks like caching is broken. It isn't. A local server has nothing to bill, so it reports 0 even while the GPU's prefix cache is working perfectly. My actual measured prefix-cache hit rate on that lane was 90.3%. The lesson: verify caching by timing the same prompt twice, not by trusting a token field. A zero from an unverified counter is a statement about the counter, not the system.

What actually breaks a cache

Exact caches only hold while the front of the prompt stays byte-for-byte identical. The things that silently destroy them are boring and fixable: a timestamp or session-id placed early in the prompt, adding or removing a tool mid-session, reordering the system prompt or tool list, or a file written into the workspace that gets read back into context. Put static content first, volatile content last, and don't mutate the tool list mid-session — each fix is provable by capturing the request before and after.

None of this needs statistics. It's counting: tokens per bucket, price per bucket, reads per write. That's the whole game.

At VigyanBytes we turn this into a product — an AI cost audit that shows a business exactly what its agents spend and where caching is helping or hurting, proven by the actual requests, not averages. If your AI bill is climbing and nobody can say why, that's a conversation worth having.

Interview-question credit to the prompt-caching breakdown by Tanishq Singh — exact vs semantic in six minutes.