What quantisation buys, exactly
Every parameter in a neural network occupies a fixed number of bits, so the weight memory is a product of two numbers and nothing else: parameters × bits ÷ 8 bytes. Quantisation reduces the second factor. Going from 16 bits to 4 divides the footprint by four, and going from 16 to 8 halves it. There is no compression algorithm involved and no data-dependent ratio to estimate — the saving is arithmetic.
That certainty is what makes quantisation the first tool to reach for when a model does not fit. A 70-billion-parameter model at BF16 needs 140 GB and requires at least two 80 GB accelerators. The same model at 4 bits needs about 35 GB and fits on one. The hardware bill roughly halves, and so does the fixed cost in any hosting comparison.
Two things complicate the clean division, and both are in this calculator because leaving them out is how estimates come in optimistic. Not every weight is quantised. Embedding tables, layer-norm parameters and the output projection are commonly left at higher precision because they are disproportionately sensitive, and they are a real share of a smaller model's parameters. Quantised weights need metadata. Integer formats store a scale, and often a zero-point, per group of weights, so the true cost is the nominal bit width plus the metadata amortised across the group.
Put the two together and a scheme described as "4-bit" typically lands somewhere above 4 effective bits per weight. The calculator reports that effective figure explicitly, because it is what actually determines whether the model fits.
What quantisation does not touch is the key-value cache, which scales with context length and batch size rather than with parameter count. On a long-context deployment it can rival the weights. Size it with the KV cache size calculator before concluding you have room.
The formula, and where the extra bits come from
The base relation is S = N × b ÷ 8, with N in parameters and S in bytes. With N in billions and S in decimal gigabytes it simplifies to S = N × b ÷ 8 directly: 70 billion parameters at 16 bits is 70 × 16 ÷ 8 = 140 GB.
The quantised share. If a fraction q of weights moves to the target precision and the rest stays at the source precision, the effective bit width is a weighted average. At 95% quantised to 4 bits with 5% left at 16, the unquantised remainder alone contributes 0.05 × 16 = 0.8 bits per weight — more than the metadata overhead in most schemes, and easily overlooked.
The scale overhead. Integer quantisation maps a group of real-valued weights onto integers using a scale and, in asymmetric schemes, a zero-point. Those are stored per group, not per weight, so the cost per weight is (scale bits + zero-point bits) ÷ group size. A 16-bit scale and a 16-bit zero-point over a group of 64 weights adds 32 ÷ 64 = 0.5 bits per weight. Over a group of 128 it adds 0.25. Smaller groups track the local distribution of weights more closely and cost more memory — that is the entire trade-off behind group size.
Combining them:
b_eff = q × (b_target + overhead) + (1 − q) × b_source
and the compression ratio is b_source ÷ b_eff, which is what you should quote rather than the nominal ratio of bit widths. In the default configuration — 95% of weights at 4 bits with a 0.5-bit overhead, 5% left at 16 — the effective width is 5.075 bits and the ratio is 16 ÷ 5.075 = 3.15×, not 4×.
To go the other way and work out the parameter count for an architecture you are designing, use the transformer parameter count calculator.
Worked example: a 70B model from BF16 to 4-bit
You have a 70-billion-parameter model published at BF16 and a single 48 GB GPU. The quantisation scheme you plan to use converts 95% of the weights to 4 bits, leaves embeddings and normalisation at 16, and uses a 16-bit scale and 16-bit zero-point over groups of 64, giving 32 ÷ 64 = 0.5 bits per weight of metadata.
- Size at source. 70 × 16 ÷ 8 = 140 GB. Far beyond a 48 GB card.
- Quantised portion. 0.95 × (4 + 0.5) = 0.95 × 4.5 = 4.275 bits per weight.
- Unquantised portion. 0.05 × 16 = 0.8 bits per weight.
- Effective bits. 4.275 + 0.8 = 5.075 bits per weight.
- Size at target. 70 × 5.075 ÷ 8 = 355.25 ÷ 8 = 44.40625 GB.
- Memory freed. 140 − 44.40625 = 95.59 GB.
- Compression ratio. 16 ÷ 5.075 = 3.153×, against the 4× the nominal bit width suggests.
- Headroom. 48 − 44.41 = 3.59 GB left on the card.
It fits — barely. And 3.59 GB is not enough headroom to serve anything useful: at the Llama-3-70B geometry the KV cache costs about 320 KiB per token, so 3.59 GB holds roughly 11,000 tokens in total across all concurrent sequences, before activations and framework overhead take their share. The honest conclusion is that this configuration loads on one 48 GB card and serves one short conversation.
Change the scheme to a group size of 128 (overhead 0.25 bits) and quantise 98% of weights: 0.98 × 4.25 + 0.02 × 16 = 4.165 + 0.32 = 4.485 bits, giving 70 × 4.485 ÷ 8 = 39.24 GB and 8.76 GB of headroom. Same nominal precision, 5 GB more room, purely from metadata and coverage decisions.
How to read the result
Effective bits per weight is the number that matters. It is the honest description of the scheme, it determines the footprint exactly, and it is comparable across formats in a way that a format name is not. When you see two schemes both described as 4-bit, compare their effective bit widths and you will usually find they differ.
Headroom decides whether the configuration is usable, not whether it loads. Fitting the weights is necessary and nowhere near sufficient. The KV cache, the activations of the forward pass, communication buffers and allocator fragmentation all come out of what is left. A configuration with 5% headroom loads and then fails under the first concurrent request.
Memory saved translates into hardware count, which is where the money is. The useful question is not "how many gigabytes did I free" but "did I cross a boundary" — from two GPUs to one, from an 80 GB card to a 48 GB one, from a rented cluster to a workstation. Savings that do not cross a boundary change nothing about the bill. Feed the result into the GPU hours cost calculator to price the difference.
Expect a speed gain as well as a memory gain. Token generation is bound by memory bandwidth, and each decode step reads the whole model, so halving the bytes read roughly doubles the ceiling on tokens per second. The throughput calculator quantifies it. Whether the full gain is realised depends on the kernels: dequantisation costs arithmetic, and a format without an efficient kernel for your hardware can be smaller without being faster.
The calculator says nothing about quality. Memory is exactly predictable from bit width; accuracy is not. The only reliable way to know what a quantisation costs you is to evaluate the quantised model on your own task, with your own prompts, against the unquantised baseline.
Weight memory by parameter count and precision
| Parameters | FP32 (32 bits) | FP16 (16 bits) | INT8 (8 bits) | INT4 (4 bits) |
|---|---|---|---|---|
| 7B | 28 GB | 14 GB | 7 GB | 3.5 GB |
| 13B | 52 GB | 26 GB | 13 GB | 6.5 GB |
| 34B | 136 GB | 68 GB | 34 GB | 17 GB |
| 70B | 280 GB | 140 GB | 70 GB | 35 GB |
| 405B | 1,620 GB | 810 GB | 405 GB | 202.5 GB |
Read across a row to see which precision crosses a hardware boundary. A 70B model needs two 80 GB accelerators at FP16, one at INT8, and fits a single 48 GB card at INT4 only if the scheme's effective width stays near 4 bits.
Group size is the dial behind the overhead
Quantisation error depends on how well one scale represents a group of weights, so smaller groups track the local distribution better and lose less accuracy. They also store more scales. With a 16-bit scale and a 16-bit zero-point, a group of 32 costs 1.0 extra bit per weight, a group of 64 costs 0.5, a group of 128 costs 0.25 and a group of 256 costs 0.125. On a nominally 4-bit format, moving from group 32 to group 128 cuts effective width from 5 bits to 4.25 — a 15% memory difference decided entirely by a configuration flag. Symmetric schemes that drop the zero-point halve these numbers again.
What this calculation does not tell you
- Whether quality holds. Bit width predicts memory exactly and accuracy not at all. Evaluate on your own task; the degradation is model-specific and task-specific.
- Whether the format has a fast kernel. A layout with no optimised kernel for your GPU may be smaller and slower, because dequantisation runs in the inner loop.
- Whether training or fine-tuning still works. Quantised weights are for inference. Fine-tuning approaches that attach adapters to a quantised base keep the base frozen precisely because you cannot back-propagate into 4-bit integers directly.
- The KV cache. It is a separate term that grows with context and batch, and it is quantised separately.
- Activation memory. Intermediate tensors during the forward pass need room too, and they scale with batch size and hidden dimension.
- Whether the checkpoint downloads at the target size. Some formats store both quantised weights and metadata in a container that also carries the original tensors. Check the file, not the label.
Where quantisation sits among the ways to make a model fit
When a model does not fit, four options are available and they are not equivalent.
Quantise. The saving is exact, the work is a one-off conversion, and the quality cost is real but usually modest down to 4 bits. It is the first thing to try, and often the only thing needed.
Add GPUs and shard. Tensor parallelism splits each layer across devices, so memory adds up and so does bandwidth, which helps speed as well. It costs linearly in hardware and adds a communication step per layer that shows up as reduced efficiency.
Use a smaller model. A well-chosen smaller model at higher precision is frequently better than a large model crushed to 2 bits, and it is cheaper on every axis. Compare candidates on your task rather than on parameter count.
Offload to host memory. Keeping some layers in CPU RAM lets an oversized model run at all, at a speed penalty severe enough that it suits batch work and not interactive serving, because host memory bandwidth is an order of magnitude below GPU memory bandwidth.
Whichever you choose, size the whole deployment rather than the weights alone. Weights plus KV cache plus activations is the real requirement — the GPU VRAM requirement calculator assembles all three — and the hosting economics that follow from the GPU count are what the self-hosted versus API break-even calculator turns into a monthly figure.