What “8B parameters” is actually counting
A parameter is a single learned number. The count is the sum of the sizes of every weight matrix and every normalisation vector in the network, and it is fully determined by the architecture — no training required, no checkpoint to inspect. If you can read six numbers out of a config.json, you can reproduce the figure on the model card exactly.
The count matters for three practical reasons. It sets the memory the weights occupy, at a known number of bytes each. It sets the arithmetic cost of a forward pass, which is close to two floating-point operations per parameter per token. And it is the axis every scaling-law result is stated along, so comparing two architectures fairly means comparing them at equal parameter count rather than equal layer count or equal width.
Almost all of the count lives in four places: the attention projections, the feed-forward matrices, the embedding table, and — negligibly — the normalisation vectors. Modern decoder-only models drop biases entirely and use rotary position embeddings, which are computed rather than learned, so neither contributes. That is why a formula this short reproduces published totals to the parameter.
Term by term through one transformer block
Attention. Each block holds four projections. The query matrix maps the hidden size to heads × head_dim, which equals d × d because head_dim is defined as d ÷ heads. The output projection maps back the same way, giving another d². The key and value matrices map d to kv_heads × head_dim, which under classic multi-head attention is also d × d each — hence the familiar 4d² — but under grouped-query attention is smaller in proportion to kv_heads ÷ heads.
Feed-forward. A classic transformer expands with one d × d_ff matrix and contracts with one d_ff × d matrix: 2·d·d_ff. A gated design (SwiGLU, GeGLU) adds a third matrix of the same shape for the gate: 3·d·d_ff. Because the gate costs 50% more per unit of width, gated models compensate by choosing d_ff around 8/3 × d instead of the classic 4 × d, which lands on a similar total. Llama-3-8B uses 14,336 against a hidden size of 4,096, a ratio of 3.5.
Normalisation. Two RMSNorm scale vectors per block, one before attention and one before the MLP, plus one at the end of the stack: 2d per layer and d overall. On an 8B model that is 262,144 parameters out of eight billion — 0.003% — so it never changes an answer, but leaving it out means your total does not match the model card exactly.
Embeddings. One V × d table maps tokens to vectors. If the output head is untied, a second V × d matrix maps vectors back to token logits. With a 128,256-token vocabulary and a 4,096 hidden size that is 525 million parameters per copy, so the tying decision alone moves an 8B model by more than half a billion parameters.
Worked example: counting an 8B model from its config
Take hidden size 4,096, 32 layers, feed-forward dimension 14,336, 32 attention heads, 8 key/value heads, a 128,256-token vocabulary, a gated MLP and an untied output head.
- Head dimension. 4,096 ÷ 32 = 128.
- Attention per layer. The Q and O matrices are 4,096² = 16,777,216 each, so 33,554,432 together. The K and V matrices are 4,096 × (8 × 128) = 4,096 × 1,024 = 4,194,304 each, so 8,388,608 together. Total 41,943,040.
- MLP per layer. Gated, so three matrices of 4,096 × 14,336 = 58,720,256 each: 176,160,768.
- Norms per layer. 2 × 4,096 = 8,192.
- Per layer. 41,943,040 + 176,160,768 + 8,192 = 218,112,000.
- All layers. 218,112,000 × 32 = 6,979,584,000.
- Embeddings. 128,256 × 4,096 = 525,336,576 per copy, doubled because the head is untied: 1,050,673,152.
- Total. 6,979,584,000 + 1,050,673,152 + 4,096 (the final norm) = 8,030,261,248, which is the 8.03B the model card reports.
Two comparisons fall straight out of that arithmetic. Grouped-query attention is doing real work: with 32 KV heads instead of 8, the K and V matrices would be 16,777,216 each rather than 4,194,304, adding 25,165,824 per layer and 805,306,368 across the stack — the model would be 8.84B instead of 8.03B. And the feed-forward block is 176,160,768 ÷ 218,112,000 = 80.8% of every layer, which is why depth and width changes move the count almost entirely through the MLP.
Reading the breakdown
The share that sits in embeddings is the most informative number for small models and the least for large ones. Embedding parameters scale with V × d, while layer parameters scale with L × d² — so as a model gets wider and deeper the layers grow quadratically and the embeddings only linearly. In GPT-2 small the embedding table is 31% of the model; in the 8B example it is 13%; on a 70B model with the same vocabulary and a hidden size of 8,192 it is about 3%. That is why scaling-law papers quote non-embedding parameters: the embedding table is a lookup, not computation, and including it distorts comparisons at the small end.
Within the layers, expect the feed-forward block to hold roughly three quarters to four fifths of the parameters in any modern configuration, and attention the rest. If your breakdown says otherwise, check d_ff: a ratio far from 4× (non-gated) or 8/3× to 3.5× (gated) usually means the config value was misread.
Watch what grouped-query attention does and does not do. It shrinks only the K and V projections, so on the 8B example it removes 10% of the total parameters. Its real payoff is elsewhere: the KV cache shrinks by the full kv_heads ÷ heads ratio, a factor of four here, which is the difference between long-context serving being feasible and not. Judge GQA by the cache, sized with the VRAM calculator, rather than by the parameter saving.
Finally, treat the file-size outputs as the weights only. A safetensors checkpoint on disk is close to this figure; the memory a running process needs is larger, because of the cache, the workspaces and the allocator.
Two real configurations, broken down by component
| Component | GPT-2 small geometry | Share | Llama-3-8B geometry | Share |
|---|---|---|---|---|
| Attention (all layers) | 28,311,552 | 22.91% | 1,342,177,280 | 16.71% |
| Feed-forward (all layers) | 56,623,104 | 45.83% | 5,637,144,576 | 70.20% |
| Layer norms | 9,216 | 0.01% | 262,144 | 0.00% |
| Embeddings | 38,597,376 | 31.24% | 1,050,673,152 | 13.08% |
| Final norm | 768 | 0.00% | 4,096 | 0.00% |
| Total | 123,551,232 | 100% | 8,030,261,248 | 100% |
The GPT-2 column omits the learned position embeddings (1,024 × 768 = 786,432) and the biases that model carries, both of which modern architectures drop; adding them back gives the familiar 124M. The embedding share falls from 31% to 13% between the two because layer parameters grow with d² while the embedding table grows with d.
What this formula deliberately omits
The count covers weight matrices and normalisation vectors for a standard decoder-only transformer. It excludes learned absolute or relative position embeddings, because rotary embeddings — used by essentially every recent open-weight model — are computed from the token index and learn nothing. It excludes biases, which most modern models drop. It assumes one uniform block repeated L times, so it does not describe mixture-of-experts models, where each layer holds many feed-forward blocks and only a few are active per token, nor encoder-decoder architectures with cross-attention. For an MoE model, multiply the feed-forward term by the number of experts to get the total parameters, and leave it as-is to get the active parameters.
Mistakes that make the count miss the model card
- Using num_attention_heads where the config means num_key_value_heads. On a GQA model this inflates the attention term by up to 4× and the total by around 10%.
- Forgetting the third MLP matrix. A gated block has gate, up and down projections. Counting two on a SwiGLU model loses about a quarter of the whole network.
- Getting the tying wrong. An untied head stores the vocabulary matrix twice. On a small model with a large vocabulary this is the single biggest term.
- Counting position embeddings on a rotary model. RoPE has no parameters; adding V × d or context × d for it invents weights that do not exist.
- Comparing total against non-embedding. Scaling-law figures are non-embedding; model cards are total. The gap is a third on a 124M model and a tenth on an 8B one.
- Reading d_ff as the expansion ratio. intermediate_size is an absolute width, not a multiplier. Entering 4 instead of 14,336 will produce a number six orders of magnitude too small.
What the parameter count tells you next
Once you have the count, two follow-on questions have short answers. The memory needed to serve or fine-tune the model comes from the count plus the KV cache and the optimizer state, which is what the LLM VRAM requirement calculator assembles; the cache term specifically is worked through by the KV cache size calculator, and it depends on the same layers and KV heads you entered here. The training cost follows the rule that a forward and backward pass over one token costs about six floating-point operations per parameter, which the training compute calculator turns into GPU-hours.
If you are choosing between architectures at a fixed budget, count both here first, then compare them on a task metric such as F1 rather than on parameters. And when the plan is to train across many devices, the ceiling on what extra hardware buys you is set by Amdahl’s law, not by the parameter count.