What training compute means and why one number governs the budget
Training compute is the total count of floating-point operations a run performs from initialisation to the final checkpoint. It is the currency of large-model training: hardware, electricity, wall-clock time and money are all downstream of it, and it is the one quantity you can estimate accurately before writing a line of training code.
The estimate works because a transformer's forward pass is dominated by dense matrix multiplications, and every one of those multiplications touches each parameter exactly once per token. A multiply-accumulate is two floating-point operations, so the forward pass costs about 2 FLOPs per parameter per token. The backward pass computes two sets of gradients — with respect to the layer inputs and with respect to the weights — and so costs roughly twice the forward pass. Add them and you get 6 FLOPs per parameter per token, which is where C = 6ND comes from.
What the number buys you is planning leverage. Once you know C, the rest is division: divide by what your cluster actually sustains and you have GPU-hours; divide again by the device count and you have days on the calendar; multiply by your hourly rate and you have the invoice. That chain is the whole calculator, and each division is where an optimistic assumption quietly doubles the bill.
Reading the formula: what the factor of 6 includes and what it leaves out
Take the three symbols one at a time. N is the trainable parameter count including embeddings. For a mixture-of-experts model, use the parameters active per token rather than the checkpoint total, because a router that sends each token to two of sixty-four experts only performs arithmetic against those two. Sizing N from a layer configuration is what the transformer parameter count calculator does.
D is tokens processed across the entire run, not corpus size. Two epochs over a 500B-token corpus is D = 1T. Discarded tokens from packing, padding and dropped batches still cost compute, so a run that reports 2T tokens consumed usually did slightly more arithmetic than 2T implies.
The factor of 6 is an approximation with known omissions. It ignores the attention score matrix, whose cost scales with sequence length squared rather than with parameters; at a 2,048-token context that term adds a few percent, and at 128k it stops being negligible. It ignores layer norms, activation functions and the optimiser step. It also ignores recomputation: activation checkpointing trades memory for arithmetic by recomputing the forward pass during the backward pass, which pushes the true factor from 6 towards 8. If your run uses full activation checkpointing, treat this calculator's output as a floor and add roughly a third.
The second formula converts compute into time. Sustained throughput is peak throughput multiplied by model FLOPs utilisation, the ratio of useful model FLOPs to what the hardware could theoretically retire in the same window. MFU is where the honest estimate lives: it absorbs data-loading stalls, pipeline bubbles, gradient all-reduce, checkpoint writes and every failed node restart.
Worked example: GPT-3 scale, 175B parameters on 300B tokens
Take the configuration GPT-3 was trained at: 175 billion parameters, 300 billion tokens, on 1,024 accelerators rated at 312 TFLOP/s dense BF16, sustaining 40% MFU, rented at $2.00 per GPU-hour.
- Convert to raw counts. N = 175 × 109 = 1.75 × 1011. D = 300 × 109 = 3.0 × 1011.
- Apply 6ND. C = 6 × 1.75 × 1011 × 3.0 × 1011 = 6 × 5.25 × 1022 = 3.15 × 1023 FLOPs. The GPT-3 paper reports 3.14 × 1023 for the same run, so the estimate lands within a third of a percent of the published figure.
- Express it in petaflop/s-days. One petaflop/s sustained for one day is 1015 × 86,400 = 8.64 × 1019 FLOPs. So 3.15 × 1023 ÷ 8.64 × 1019 = 3,645.8 petaflop/s-days.
- Find sustained per-device throughput. 312 TFLOP/s × 0.40 = 124.8 TFLOP/s = 1.248 × 1014 FLOP/s.
- Divide for GPU-seconds. 3.15 × 1023 ÷ 1.248 × 1014 = 2.524 × 109 GPU-seconds, which is 2.524 × 109 ÷ 3,600 = 701,122 GPU-hours.
- Divide by the cluster. 701,122 ÷ 1,024 = 684.7 wall-clock hours = 28.5 days.
- Price it. 701,122 × $2.00 = $1,402,244 of compute, before storage, egress and the runs you throw away.
The Chinchilla check comes free: 300B ÷ 175B = 1.71 tokens per parameter, far below the ratio of roughly 20 that later work identified as compute-optimal. At the same 3.15 × 1023 FLOP budget, a model of about 40B parameters trained on 800B tokens would have reached a lower loss.
How to read the result: MFU, the Chinchilla ratio, and what to do with each
Read the tokens-per-parameter figure first, because it tells you whether the run is well specified before you worry about how long it takes. Hoffmann and colleagues fitted parameter count and token count jointly against a fixed compute budget and found that the loss-minimising split puts them in roughly equal proportion — about 20 tokens per parameter across the range they studied. A ratio far below 20 means you are spending compute on a model too large for the data you are giving it.
A ratio far above 20 is not an error. Chinchilla optimises training loss for a fixed training budget and says nothing about inference. Production models are routinely trained well past the compute-optimal point on purpose, because every extra training FLOP buys a permanently smaller model to serve, and serving cost accumulates for the life of the product. Weigh that trade with the self-host vs API breakeven calculator and the GPU VRAM requirement calculator, which price the model you end up with rather than the one you train.
Read MFU second. It is the single assumption that most changes the answer, and the utilisation table on this page exists to show how much: holding N and D fixed, moving from 20% to 60% MFU cuts both time and cost by two-thirds. Large published dense pre-training runs report MFU in roughly the 30–55% band — the PaLM paper reports 46.2% and the Megatron-LM scaling work reports about 52% at 3,072 A100s — so a plan built on 70% needs evidence behind it. Measure it on a pilot: run a few hundred steps, count tokens per second, multiply by 6N, divide by aggregate peak.
Finally, treat the cost figure as a floor. It prices successful GPU-time only. Real programmes carry failed runs, hyperparameter sweeps, data preprocessing, evaluation and idle reservation.
Training compute for representative model and token budgets
| Parameters (N) | Tokens (D) | Tokens per parameter | Compute (FLOPs) | Petaflop/s-days | Device-days at 400 TFLOP/s |
|---|---|---|---|---|---|
| 1B | 20B | 20 | 1.20 × 1020 | 1.39 | 3.5 |
| 7B | 140B | 20 | 5.88 × 1021 | 68.1 | 170 |
| 7B | 2T | 286 | 8.40 × 1022 | 972 | 2,431 |
| 70B | 2T | 28.6 | 8.40 × 1023 | 9,722 | 24,306 |
| 175B | 300B | 1.71 | 3.15 × 1023 | 3,646 | 9,115 |
| 405B | 15T | 37.0 | 3.65 × 1025 | 421,875 | 1,054,688 |
Device-days are aggregate: divide by your accelerator count for wall-clock time. A 24,306 device-day run finishes in 24 days on 1,024 devices, assuming nothing fails.
Mistakes that make a compute estimate wrong
- Using the sparsity-enabled peak. Datasheets lead with a number that assumes 2:4 structured sparsity, which dense pre-training does not use. It is exactly double the dense figure, so this mistake halves your estimate.
- Confusing corpus size with tokens processed. D is tokens seen, so multiple epochs multiply it. A three-epoch fine-tune on a 10B-token corpus is D = 30B.
- Ignoring activation checkpointing. Full recomputation adds an extra forward pass, moving the effective factor from 6 towards 8 — about 33% more compute for the same N and D.
- Counting all MoE parameters. Only the experts a token is routed to perform arithmetic for that token. Use active parameters, or the estimate overshoots by the sparsity factor.
- Assuming MFU is constant across scale. Utilisation usually falls as you add devices, because communication grows while per-device arithmetic does not. Estimate it at the cluster size you will actually run.
- Forgetting the long-context term. Attention costs roughly 12 × layers × d_model × sequence-length FLOPs per token beyond 6N. At 2k context this is small; at 128k it can rival the parameter term.
Where this estimate sits among the alternatives
6ND is a planning instrument, not an accounting one. If you need the actual arithmetic your run performed, instrument it: most frameworks can report tokens per second, and multiplying that by 6N gives measured model FLOPs directly. Hardware counters give a different number again — total FLOPs retired, including recomputation and padding — and the ratio between the two is precisely what MFU measures.
For fine-tuning rather than pre-training, the same formula applies with a much smaller D, but parameter-efficient methods break it. LoRA still runs the full forward and backward passes through the frozen base model, so compute barely falls even though the number of trainable parameters collapses; use full N in the formula and expect only the optimiser-state memory to shrink. The fine-tune vs prompting breakeven calculator handles that comparison, and the model training time estimate calculator works from measured steps per second when you already have a job running.
For the money question specifically, compute is only one line. Storage for checkpoints of a 405B model runs to terabytes per snapshot, interconnect provisioning is often bundled into the hourly rate, and reserved capacity bills whether or not a job is healthy. The GPU-hours cost calculator takes the aggregate hours this page produces and prices them against instance types and commitment terms.
Finally, note the regulatory angle. Several jurisdictions now use total training compute as a threshold for reporting obligations on frontier models, with 1025 and 1026 FLOPs both appearing in current rules. Because 6ND is the standard estimator, the number this page returns is the number those thresholds are written against.
Key terms
- FLOP
- One floating-point operation. Note the distinction from FLOP/s, a rate. Training compute is measured in FLOPs (a count); hardware is rated in FLOP/s (a speed).
- Petaflop/s-day
- The compute performed by a machine sustaining 1015 FLOP/s for 24 hours: 8.64 × 1019 FLOPs. Used in the GPT-3 paper and widely since.
- MFU (model FLOPs utilisation)
- Useful model FLOPs divided by the hardware's theoretical peak over the same wall-clock window. It excludes recomputation, so it is always at or below hardware FLOPs utilisation.
- Compute-optimal
- The parameter and token split that minimises loss for a fixed training compute budget. The Chinchilla result puts it near 20 tokens per parameter.
- Active parameters
- For sparse mixture-of-experts models, the parameters that participate in the forward pass for a single token. This is the N that belongs in 6ND.