What prompt caching charges you for
Prompt caching stores the model's internal state for a prefix of your request so that identical prefixes on later requests skip the work of processing those tokens again. Providers price this in three tiers rather than two: the normal input rate for uncached tokens, a higher rate on the request that writes the cache, and a much lower rate on each request that reads it.
That structure is what makes caching a break-even question rather than a free win. Writing the cache costs more than not caching at all. If the entry expires before enough requests reuse it, you have paid a premium for nothing. If many requests reuse it, the premium is spread thin and you approach the read rate, which is a fraction of what you were paying.
The calculator works in units of a cache lifetime: one write followed by however many hits occur before the entry expires. That is the correct accounting boundary, because an entry that expires after four hits and is rewritten costs the write premium again. Two hundred requests spread over a day with a five-minute time-to-live are not two hundred hits on one entry — they may be dozens of separate lifetimes, each paying a write.
What caching does not touch is everything after the prefix. The user's actual message, the retrieved documents that change per request, and the model's reply are billed exactly as before. So the saving is bounded by the share of your input that is genuinely constant, which is why the first question to answer is how many of your tokens never change.
The formula, and where the break-even comes from
Let p·b be the cost of sending the prefix once at the base rate, where p is the prefix length in millions of tokens and b the price per million. Over R requests:
Without caching: R × p × b
With caching: p × b × w + (R − 1) × p × b × k
One write at the multiplier w, then R − 1 hits at the multiplier k. Set the two equal and the prefix cost p·b cancels out entirely:
R = w + (R − 1)k → R − Rk = w − k → R* = (w − k) ÷ (1 − k)
Three things follow from that expression, and each is worth internalising.
Break-even does not depend on prefix length or price. Both cancel. A 200,000-token prefix on an expensive model and a 2,000-token prefix on a cheap one break even at the same number of requests. Length and price decide how much you save, not whether you save.
Break-even is usually small. With a 1.25× write and a 0.10× read, R* = (1.25 − 0.10) ÷ 0.90 = 1.278, so two requests inside one lifetime already put you ahead. With a 2.0× write and a 0.25× read, R* = 1.75 ÷ 0.75 = 2.333, so you need three.
The saving has a ceiling of 1 − k. As R grows, the write premium is amortised away and the cost per request tends to k times base. At a 0.10× read rate the maximum possible saving on the cached prefix is 90%, and you get most of the way there within a few dozen requests. Chasing higher reuse past that point buys very little.
The formula assumes k < 1. If your read multiplier is 1 or above there is no discount to earn, the cached path never costs less, and the calculator reports no break-even.
Worked example: a 10,000-token system prompt at $3 per million
Your agent carries a 10,000-token prefix on every call: a system prompt, a tool schema and a fixed style guide. The base input rate is $3.00 per million tokens, the cache write multiplier is 1.25× and the read multiplier is 0.10×. Traffic is dense enough that 50 requests hit each cache entry before it expires, and you serve 200,000 such requests a month.
- Cost of the prefix once. 10,000 ÷ 1,000,000 = 0.01 million tokens; 0.01 × $3.00 = $0.03 per request uncached.
- Without caching, per lifetime. 50 × $0.03 = $1.50.
- The write. $0.03 × 1.25 = $0.0375.
- The 49 hits. 49 × $0.03 × 0.10 = 49 × $0.003 = $0.147.
- With caching, per lifetime. $0.0375 + $0.147 = $0.1845.
- Saving per lifetime. $1.50 − $0.1845 = $1.3155, which is 1.3155 ÷ 1.50 = 87.7% of the prefix cost.
- Break-even. (1.25 − 0.10) ÷ 0.90 = 1.278, so 2 requests per lifetime is enough to be ahead.
- Monthly. 200,000 requests ÷ 50 per lifetime = 4,000 lifetimes; 4,000 × $1.3155 = $5,262 a month.
Check the ceiling: 87.7% against a maximum of 1 − 0.10 = 90%. The remaining 2.3 percentage points are the write premium spread across 50 requests, and pushing reuse to 100 requests would only reach 88.85%. The bulk of the win arrives in the first handful of hits.
Now the failure case. Suppose the entry expires before the second request, so every request is a write with no hits. Cost per request becomes $0.0375 against $0.03, and you have made the prompt 25% more expensive. That is the whole risk of caching, and it is entirely a question of traffic density against time-to-live.
How to read the result
Compare requests per lifetime with the break-even count first. Everything else is secondary. Above it you save, below it you pay a premium, and the calculator says which side you are on. Because break-even is typically two or three requests, most production traffic clears it easily — the cases that fail are low-volume endpoints, per-user prefixes that only one user touches, and batch jobs that run once an hour against a five-minute time-to-live.
Read the percentage against its ceiling of 1 − k, not against 100%. A saving of 87.7% where 90% is the maximum means you have captured nearly all of what is available. A saving of 40% where 90% is available means your reuse count is low and there is real room to improve it by extending the entry lifetime or routing more traffic through the same prefix.
Remember the percentage applies to the prefix only. If the prefix is 10,000 of a 12,000-token request, an 87.7% saving on the prefix is a large cut in the total bill. If the prefix is 2,000 of a 12,000-token request, the same 87.7% barely moves it. Work out the prefix share of your total input before deciding this is the lever that matters; the token cost calculator gives the full request breakdown.
If the monthly figure is negative, the fix is structural rather than numerical. Raise the number of requests that share one entry: use a single global prefix instead of a per-user one, order your prompt so the stable content comes first, and keep any per-request content strictly after the cache breakpoint. A prefix that varies by even one token is a different prefix and a different cache entry.
Saving on the cached prefix by reuse count and multiplier pair
| Requests per cache lifetime | Write 1.25× / read 0.10× | Write 2.00× / read 0.25× |
|---|---|---|
| 1 | −25.0% | −100.0% |
| 2 | 32.5% | −12.5% |
| 3 | 51.7% | 16.7% |
| 5 | 67.0% | 40.0% |
| 10 | 78.5% | 57.5% |
| 25 | 85.4% | 68.0% |
| 50 | 87.7% | 71.5% |
| 100 | 88.9% | 73.3% |
| Ceiling (R → ∞) | 90.0% | 75.0% |
Break-even is 1.28 requests in the first column and 2.33 in the second, which is why the first column is already positive at two requests and the second is not. Both curves flatten quickly: more than three quarters of the achievable saving arrives by the tenth request.
The cache key is an exact prefix match
A cache entry is found only when the beginning of your request matches the cached content exactly, token for token. Injecting the current timestamp, a request identifier, a user name or a randomised instruction anywhere in the prefix produces a different prefix and therefore a miss — and a miss means you pay the write premium again. This is the most common reason a deployment reports far fewer hits than expected. Put every stable element first, in a fixed order, and place anything that varies after the cache breakpoint. The same applies to tool definitions serialised from a hash map with non-deterministic ordering.
Mistakes that make the saving smaller than modelled
- Counting monthly requests as requests per lifetime. A cache entry has a time-to-live measured in minutes. Only requests arriving inside that window count as hits on the same entry.
- Per-user or per-session prefixes. One entry per user means the reuse count is that user's request rate, not your total traffic. Global prefixes are what deliver high reuse.
- Variable content before the stable content. Anything that changes must come after the cached prefix, or nothing behind it can be cached.
- Ignoring the minimum cacheable length. Providers set a floor on how long a prefix must be to be cacheable. A prefix below it is billed normally and the write premium may still apply to the attempt.
- Assuming the reply is cached. Caching covers input processing, not generation. Output tokens are billed at their usual rate no matter how repetitive the answer is.
- Forgetting concurrency. Several instances warming the same prefix at once can each write the cache, multiplying the premium. Warm deliberately with one request before opening the traffic gate.
- Comparing against the wrong baseline. If you would have shortened the prompt instead, the fair comparison is caching against the shorter prompt, not against the long one you kept.
Where caching sits among the other cost levers
Caching is the cheapest of the major cost levers to implement, which is why it belongs first in any optimisation sequence. It requires no change to model behaviour, no evaluation work and usually no more than reordering a prompt and adding a breakpoint. The savings arrive on the same day.
The next lever is shortening the context itself. That reduces the prefix you cache and the tokens you never cache alike, and it speeds up prefill, so it improves latency as well as cost — see the throughput and latency calculator for how prompt length maps to time to first token. Caching and shortening are complements, not alternatives: cache what must stay, cut what need not.
Further along sits fine-tuning, which removes a long few-shot prompt permanently by putting the behaviour in the weights. That trade has its own break-even in requests, and it competes directly with caching, because a cached few-shot prompt is already cheap. Work out both before committing: the fine-tuning versus long-prompt calculator handles the training-cost side, and if the prompt is cached at a 0.10× read rate the case for fine-tuning is considerably weaker than the uncached comparison suggests.
Finally, translate whatever you save into the metric your business actually reports. A monthly saving is an infrastructure number; gross margin per user is the one that decides pricing. Feed the reduced blended rate into the cost per user per month calculator to see how far the saving moves it.