The four things that occupy a GPU when you run a model
People size GPUs by parameter count alone, and then run out of memory at 3,000 tokens of context. Parameter count sets only the first of four terms. What actually has to fit is the weights, the KV cache that grows with every token you process, the gradient and optimizer state if you are training, and a layer of framework overhead that is real, unglamorous and consistently forgotten.
The weights are the easy part: multiply the parameter count by the bytes per weight. A bf16 model needs two bytes per parameter, so 8 billion parameters is 16 billion bytes, which is 14.90 GiB once you divide by 1024³. That division is the first place estimates go wrong — GPU memory is quoted in GiB by every tool that reports it, while model sizes are quoted in decimal billions, and the two differ by 7.4%.
The KV cache is the term that surprises people. During generation the model keeps the key and value vectors for every token it has already seen, in every layer, so it does not have to recompute attention over the whole prefix at each step. That is a per-token, per-sequence cost, and at long context it can exceed the weights. Training replaces this term with a much larger one: gradients, Adam moments and master weights, which together run to 14 bytes per trainable parameter.
Where each term comes from
Weights. Parameter count times bytes per parameter. Quantisation is exactly linear here: int8 halves fp16, int4 halves int8. Nothing else in the budget shrinks when you quantise the weights, which is why the total never falls as fast as the headline claim.
KV cache. Two tensors (a key and a value) per token, per layer, per key/value head, each of head-dimension length: 2 × layers × kv_heads × head_dim × tokens × sequences × bytes. The kv_heads figure is the one to check in the config. Under multi-head attention it equals the number of attention heads; under grouped-query attention, which nearly every recent open-weight model uses, it is much smaller — 8 instead of 32 on Llama-3-8B, cutting the cache by a factor of four. That reduction is the main reason GQA exists.
Training state. Mixed-precision training with Adam keeps, per trainable parameter, a 2-byte gradient, a 4-byte fp32 master copy of the weight, and two 4-byte Adam moments — 14 bytes on top of the 2-byte working weight, so 16 bytes per parameter in total. That is why a full fine-tune of a 7B model needs roughly eight times the memory of serving it, and why LoRA works: freeze the base, train adapters worth well under 1% of the parameters, and the 14-byte term applies only to them.
Activations and overhead. The backward pass needs activations from the forward pass. With gradient checkpointing you keep roughly one hidden-state tensor per layer, which is batch × tokens × hidden × layers × 2 bytes; without it the figure is several times larger and depends on the implementation. On top of that sits the CUDA context, kernel workspaces and allocator fragmentation, budgeted here as a percentage of the weights — 10–20% is a workable allowance for a typical serving stack.
Worked example: serving an 8B model at 8k context
Take a Llama-3-8B-class model: 8.03 billion parameters, 32 layers, hidden size 4,096, 8 key/value heads, head dimension 128. You want to serve it in bf16 at 8,192 tokens of context, one request at a time, on a single 80 GiB card.
- Weights. 8 × 10⁹ × 2 bytes = 16 × 10⁹ bytes. Divide by 1024³ = 1,073,741,824: 14.901 GiB.
- KV cache per token. 2 × 32 layers × 8 kv heads × 128 × 2 bytes = 131,072 bytes = 128 KiB per token.
- KV cache total. 131,072 × 8,192 tokens × 1 sequence = 1,073,741,824 bytes = exactly 1.000 GiB.
- Overhead. 15% of the weights = 0.15 × 14.901 = 2.235 GiB.
- Total. 14.901 + 1.000 + 2.235 = 18.136 GiB, so one 80 GiB card with 61.86 GiB to spare.
That headroom is not wasted — it is what lets you raise the batch size. Each additional concurrent sequence at 8k context costs another 1.000 GiB, so the same card holds roughly 60 concurrent 8k sequences before memory, rather than compute, becomes the limit.
Push the context instead and the arithmetic changes character. At 131,072 tokens the cache is 16 × 1.000 = 16.000 GiB, slightly more than the weights themselves, and the total reaches 33.136 GiB. At that point quantising the weights to int4 would save 11.18 GiB while halving the KV cache precision would save 8 GiB — and the second option leaves the model’s output quality alone in a way the first does not.
Reading the result and deciding what to change
Compare the total against usable memory, not marketed memory. A card sold as 80 GB reports about 79.2 GiB, the driver and CUDA context hold some of that before your process allocates anything, and PyTorch’s caching allocator fragments over long-running serving. Leaving 10% clear of the number this calculator produces is a reasonable operating margin; running at 98% will work in a benchmark and fail in production.
Read the component breakdown before you decide what to cut, because the biggest term tells you which lever works. If the weights dominate, quantisation helps and context reduction does almost nothing. If the KV cache dominates — which the calculator flags explicitly when it exceeds the weights — quantising further is close to pointless, and the effective moves are a shorter context, a smaller batch, an fp8 cache, or a model with fewer KV heads.
When the answer is more than one GPU, treat the number as a floor. Tensor parallelism replicates some buffers and adds communication workspace on every device, so a model that computes to 1.4 GPUs does not run on two GPUs at 70% each. Pipeline parallelism adds activation buffers between stages. In both cases the practical requirement lands above the arithmetic.
For training, the single most useful comparison is LoRA against a full fine-tune on the same model. At 0.5% trainable parameters, the 14-byte term applies to 1/200th of the network, so the gradient and optimizer memory falls by the same factor — on a 7B model that is roughly 98 GiB down to 0.5 GiB. The frozen base still has to be resident, so LoRA does not make the weights free, but it removes the term that made full fine-tuning impossible on one card.
Weight memory alone, by model size and precision
| Parameters | fp32 (4 B) | fp16 / bf16 (2 B) | int8 (1 B) | int4 (0.5 B) |
|---|---|---|---|---|
| 1B | 3.73 GiB | 1.86 GiB | 0.93 GiB | 0.47 GiB |
| 7B | 26.08 GiB | 13.04 GiB | 6.52 GiB | 3.26 GiB |
| 8B | 29.80 GiB | 14.90 GiB | 7.45 GiB | 3.73 GiB |
| 13B | 48.43 GiB | 24.21 GiB | 12.11 GiB | 6.05 GiB |
| 70B | 260.77 GiB | 130.39 GiB | 65.19 GiB | 32.60 GiB |
| 405B | 1,508.71 GiB | 754.35 GiB | 377.18 GiB | 188.59 GiB |
Every row is a multiple of the 1B row, because the relationship is exactly linear. The 70B int4 row at 32.60 GiB is why two 24 GiB consumer cards are the usual floor for running a 70B model locally.
GB and GiB are not the same, and it matters here
Model sizes are quoted in decimal billions of parameters, and GPU memory is reported in binary gibibytes. One GiB is 1,073,741,824 bytes, 7.4% more than a decimal gigabyte. An 8B bf16 model is 16 decimal GB of weights and 14.90 GiB — the same quantity described two ways. This calculator works in GiB throughout, because that is what nvidia-smi, PyTorch and every allocator error message report. If you compare a GiB figure against a GB card specification you will conclude a model fits when it does not.
Assumptions this calculator makes, and what it leaves out
- It assumes gradient checkpointing when training. Without it, retained activations are several times larger and depend heavily on the implementation and on whether attention is computed with a fused kernel.
- It assumes Adam or AdamW. SGD with momentum keeps far less state; 8-bit optimizers cut the moment tensors to a quarter of the fp32 figure.
- It assumes one device holds everything. ZeRO stage 2 and 3, and FSDP, shard the optimizer states, gradients and eventually the parameters across devices, dividing the largest training term rather than replicating it.
- It assumes a dense model. A mixture-of-experts model must hold all expert weights in memory even though only a few are active per token, so size it by total parameters, not active ones.
- It ignores paged and shared caches. Serving stacks that page the KV cache or share prefixes across requests use less than the per-sequence arithmetic suggests when prompts overlap.
- It ignores speculative decoding and draft models. A draft model is a second set of weights and a second cache, both of which have to fit alongside the target.
- It says nothing about speed. Fitting in memory and running fast are separate questions; memory bandwidth, not capacity, usually sets tokens per second once the model fits.
What to work out next
If you do not know the parameter count — or you want to know how it splits between attention, feed-forward and embeddings — derive it from the config with the transformer parameter count calculator, then bring the total back here. For a closer look at the term that dominates long-context serving, the KV cache size calculator handles paged allocation and per-request budgeting in more detail than the single line in this budget.
Memory is only one of the two constraints on a training run; the other is arithmetic, which the training compute calculator estimates from tokens and parameters. When you are sizing a multi-GPU job, the scaling you actually get is bounded by the serial fraction of the work, which is the question Amdahl’s law answers. And once the model is trained and served, whether it is any good is a different measurement entirely — that is what the F1 and classification metrics calculator is for.