What the KV cache is and why it exists
Self-attention lets every token attend to every token before it. Computing that from scratch at each decode step would mean re-deriving the key and value vectors for the entire sequence on every single token — quadratic work repeated over and over. The key-value cache avoids it: once a token's keys and values are computed, they never change, so they are stored and reused. The step that produces token 5,000 reads 4,999 cached entries and appends one.
The trade is memory for compute, and it is overwhelmingly worth making. But the memory is not small, and unlike the weights it is not fixed. Weights are loaded once and occupy the same space forever. The cache grows with every token in every concurrent sequence, so it is the term that decides how many users you can serve at once and how long a conversation can get before the deployment falls over.
The size is exactly predictable, which is what makes this calculation useful. For each layer the model stores one key vector and one value vector per KV head per token, each of head-dimension width, at whatever precision the cache is kept in. Multiply those out and you get bytes per token — a constant for a given architecture. Everything else is multiplication by context length and batch size.
The consequence worth internalising: the cache is linear in context length and linear in batch size. Doubling either doubles it. A deployment sized comfortably at 4k context with 16 concurrent users is out of memory at 32k context with the same 16 users, and nothing about the model has changed.
The formula, and why query heads are absent
bytes = 2 × L × h_kv × d_h × s × b × p
The leading 2 is keys and values — two tensors of identical shape per layer. It is not a fudge factor, and dropping it is the most common error in a hand estimate.
L, layers. Each decoder block runs its own attention and keeps its own cache, so the cache scales with depth. This is why a deep 70B model with few KV heads can still hold a substantial cache.
h_kv, key/value heads. The term that architecture design has attacked hardest. In classic multi-head attention every query head has its own key and value head, so h_kv equals the head count. Grouped-query attention shares one KV head across a group of query heads, and multi-query attention takes it to the limit with a single KV head for the whole layer. Llama-3-70B has 64 query heads and 8 KV heads, so its cache is eight times smaller than the same model with full multi-head attention would need.
Query heads are not in the formula at all. Queries are computed fresh for the single new token at each step and thrown away — there is nothing to cache. That asymmetry is precisely what grouped-query attention exploits: you keep the expressive power of many query heads while paying cache for only a few KV heads.
d_h, head dimension. Usually hidden size divided by query head count; 128 is the common choice across current models.
p, bytes per element. FP16 and BF16 give 2. The cache can be quantised independently of the weights, and an 8-bit cache halves this term while leaving model quality largely governed by the weights' precision.
To size the weights alongside the cache, use the GPU VRAM requirement calculator; to see what different weight precisions cost, use the quantisation memory savings calculator.
Worked example: a 70B model at 8k context on two 80 GiB GPUs
Take the Llama-3-70B geometry: 80 layers, 64 query heads, 8 KV heads, head dimension 128. Serve it at BF16 with an FP16 cache, at 8,192 tokens of context, on two 80 GiB GPUs (160 GiB total) where the weights occupy about 131 GiB.
- Bytes per token. 2 × 80 × 8 × 128 = 163,840 elements per token; × 2 bytes = 327,680 bytes.
- In KiB. 327,680 ÷ 1,024 = 320 KiB per token.
- Per sequence. 327,680 × 8,192 = 2,684,354,560 bytes; ÷ 1,073,741,824 = 2.5 GiB for one full-length sequence.
- Memory left after weights. 160 − 131 = 29 GiB.
- Sequences that fit. 29 ÷ 2.5 = 11.6, so 11 concurrent sequences at full context.
- At batch 8. 8 × 2.5 = 20 GiB, which is 20 ÷ 160 = 12.5% of total GPU memory.
Now change one thing at a time and watch what happens.
Extend to 32k context. Per sequence becomes 4 × 2.5 = 10 GiB, and 29 ÷ 10 = 2 sequences fit. You have gone from 11 concurrent users to 2 without touching the model.
Quantise the cache to 8 bits at 32k. Per sequence halves to 5 GiB, and 29 ÷ 5 = 5 sequences fit. One bit-width decision more than doubles concurrency.
Imagine the same model with multi-head attention — 64 KV heads instead of 8. Bytes per token would be 327,680 × 8 = 2,621,440, so one 8k sequence would need 20 GiB and only one sequence would fit in the 29 GiB free. Grouped-query attention is the reason this deployment is viable at all.
How to read the result
Per-token KiB is the architecture constant to remember. It does not depend on your workload, so it is the number to carry around: 320 KiB per token for a 70B-class grouped-query model at FP16, 128 KiB for an 8B one. Multiply by context and batch and you have the answer without a calculator.
Sequences that fit is the capacity number. It sets your maximum concurrency, which in turn sets aggregate throughput, because throughput scales with batch size while decode remains memory-bound — see the throughput and latency calculator. A cache that squeezes batch size down to two or three sequences caps your tokens per second regardless of how fast the GPU is.
Share of GPU memory tells you which term is your ceiling. Below roughly a third, the weights dominate and quantising them is the productive move. Above half, the cache dominates and the levers are context length, batch size and cache precision. Applying a weight-quantisation fix to a cache-bound deployment frees memory that the cache immediately consumes.
Treat the fit number as an upper bound. Activations during the forward pass, the CUDA context, the framework's own buffers and allocator fragmentation all take memory this calculation ignores, and a serving stack that pre-allocates the cache in fixed blocks rounds each sequence up to a block boundary. Leaving 10–15% of memory unallocated is ordinary practice, and the calculator's arithmetic limit should be discounted accordingly.
One optimistic assumption is built in: every sequence is charged at full context. Real traffic has a distribution of lengths, and a paged cache allocator only materialises the blocks a sequence actually uses, so measured concurrency is often well above this figure. Size for the worst case and let paging give you the upside.
KV cache per token for common published architectures
| Model geometry | Layers | KV heads | Head dim | Per token | Per 8k sequence |
|---|---|---|---|---|---|
| Llama-2-7B (multi-head) | 32 | 32 | 128 | 512 KiB | 4.0 GiB |
| Llama-3-8B (grouped-query) | 32 | 8 | 128 | 128 KiB | 1.0 GiB |
| Mistral-7B (grouped-query) | 32 | 8 | 128 | 128 KiB | 1.0 GiB |
| Llama-3-70B (grouped-query) | 80 | 8 | 128 | 320 KiB | 2.5 GiB |
| GPT-3 175B (multi-head) | 96 | 96 | 128 | 4,608 KiB | 36.0 GiB |
Compare the first two rows: both are 7–8B models with 32 layers, and moving from 32 KV heads to 8 divides the cache by four. The last row is why long context on a dense multi-head model of that size was impractical without architectural change.
Binary gigabytes, and why the numbers come out round
This calculator works in GiB (1,073,741,824 bytes) rather than decimal GB (1,000,000,000 bytes), because memory allocation is binary and the arithmetic then produces exact figures — 320 KiB per token times 8,192 tokens is exactly 2.5 GiB, with no rounding anywhere. GPU marketing figures are usually decimal and the usable memory reported by the driver is lower still, since the CUDA context and framework reserve some before your model sees any. Enter the memory your allocator actually reports as available, not the number on the product page.
Mistakes that make a cache estimate wrong
- Using query heads instead of KV heads. On a grouped-query model this overstates the cache by the group size — eight times for Llama-3-70B. Read the KV head count off the model config, not the head count.
- Forgetting the factor of 2 for keys and values. Halves the estimate, and the result still looks plausible.
- Charging the batch at average length instead of maximum. A sequence's cache grows as it generates. Size for the maximum context you will admit, or you will fail at the tail.
- Ignoring the weights when computing what fits. Free memory is total minus weights minus framework overhead, not total.
- Assuming the cache precision follows the weight precision. They are independent settings in most serving stacks, and the cache is often left at FP16 while the weights are quantised to 4 bits.
- Overlooking speculative decoding and beam search. Both hold more than one candidate continuation, and each candidate carries its own cache entries.
- Forgetting that a shared prompt prefix can be shared in cache too. Serving stacks that deduplicate identical prefixes across sequences store them once, which makes the real usage lower than this arithmetic for workloads with a common system prompt.
How the cache is being attacked, and what to do about it
Nearly every recent efficiency technique in transformer serving is aimed at this one term, which tells you how binding it is.
Architectural sharing came first. Multi-query attention collapses the cache to a single KV head per layer; grouped-query attention keeps a small number of them and recovers most of the quality that multi-query gives up. Both are decisions made at training time, so as a practitioner you choose them by choosing a model — and the KV head count is worth checking before you commit to serving something at long context.
Cache quantisation works after training. Storing keys and values at 8 bits instead of 16 halves the term outright, and the effect on output quality is generally far smaller than quantising the weights by the same amount, because the cache holds activations rather than learned parameters. It is the single highest-leverage setting available at serving time.
Paged allocation attacks waste rather than size. Instead of reserving the maximum context for every sequence, a paged allocator hands out fixed-size blocks on demand and shares identical prefix blocks between sequences. It does not change bytes per token; it changes how many of those bytes you allocate but never use, which in practice is a large share.
Shorter context remains the blunt instrument that always works. Retrieval that returns three well-chosen chunks instead of twenty mediocre ones reduces the cache, the prefill time and the token bill simultaneously. Size that trade with the RAG chunking calculator, and see what the token side is worth with the prompt caching savings calculator.
Finally, connect the memory number to money. Concurrency limits set throughput, throughput sets how many GPUs you need, and GPU count sets the bill — which the self-hosted versus API break-even calculator turns into a cost per million tokens. A cache decision made here propagates all the way to that figure.