What governs how long a training job takes
Training time is an arithmetic problem with one measured input. The dataset size, batch size and epoch count determine exactly how many optimiser steps the job performs — that part is counting, not estimating. Everything uncertain collapses into a single number: how many steps per second one device sustains on your model at your batch size. Measure that once and the rest is division.
The step count is a ceiling, not a quotient, because the final batch of each epoch is usually partial and still costs a full step. With 1,000,000 samples and an effective batch of 256, an epoch is ceil(1,000,000 ÷ 256) = 3,907 steps, not 3,906.25. On a short job with many epochs this rounding is invisible; on a job with 200 epochs over a small dataset it adds up.
The second uncertainty is what happens when you add devices. In data parallelism each device holds a complete copy of the model, processes its own micro-batch, and then every device exchanges gradients before the optimiser update. That exchange is the tax. If it is fully overlapped with computation, the step rate is unchanged and eight devices process eight times the samples per step — a perfect eightfold speed-up. If the exchange is exposed, the step takes longer and the speed-up falls short. Scaling efficiency is the one number that captures the difference.
The formula, term by term, and why efficiency divides rather than multiplies
Start with the numerator. S ÷ (b · G) is samples divided by the effective batch, so it counts steps in one epoch; the ceiling covers the partial final batch; multiplying by E gives total steps. Nothing here depends on hardware.
The denominator is where hardware enters. f is the step rate you measured on one device at per-device batch b. Note the qualifier carefully: f must be measured at the same per-device batch you enter above. A step rate measured at batch 8 does not describe a job running at batch 64; larger micro-batches usually raise samples per second while lowering steps per second, so mixing the two produces an estimate that is wrong by roughly the batch ratio.
η is scaling efficiency: the step rate the whole cluster achieves divided by the step rate one device achieves. It divides the rate, which is why it multiplies the time. At η = 0.85, the job takes 1 ÷ 0.85 = 1.176 times as long as a perfectly scaling job with the same step count, and because you are paying for all G devices for that whole duration, the device-hours rise by the same 17.6%.
The speed-up figure reported above compares two complete runs rather than assuming G · η: it recomputes the step count for a single device (a much larger number, since the effective batch is G times smaller) and divides the two wall-clock times. That is why the reported speed-up is not exactly 8 × 0.85 = 6.80 in every case — the two ceilings round differently.
The scaling model here is deliberately simple, and it is the data-parallel analogue of Amdahl's argument: a fixed serial fraction of each step caps the achievable speed-up no matter how many devices you add. If you want the general form of that limit, the Amdahl's law speed-up calculator derives it.
Worked example: 1,000,000 samples on 8 GPUs for 3 epochs
You are fine-tuning on 1,000,000 samples with a per-device batch of 32, on 8 accelerators, for 3 epochs. A pilot run on one device sustains 4.0 optimiser steps per second at batch 32. Your cluster achieves 85% scaling efficiency, and the devices cost $2.50 each per hour.
- Effective batch. 32 × 8 = 256 samples per optimiser step.
- Steps per epoch. 1,000,000 ÷ 256 = 3,906.25, and the partial batch still costs a step, so ceil(3,906.25) = 3,907 steps.
- Total steps. 3,907 × 3 = 11,721 steps.
- Achieved step rate. 4.0 × 0.85 = 3.4 steps per second.
- Wall-clock time. 11,721 ÷ 3.4 = 3,447.35 seconds = 57.5 minutes = 0.958 hours.
- Throughput. 256 × 3.4 = 870.4 samples per second.
- Cost. 0.958 h × 8 devices × $2.50 = $19.15.
Now compare against one device. There the effective batch is 32, so an epoch is ceil(1,000,000 ÷ 32) = 31,250 steps, three epochs is 93,750 steps, and at 4.0 steps per second that is 23,437.5 seconds = 6.510 hours. The speed-up is 23,437.5 ÷ 3,447.35 = 6.80× on eight devices.
The cost comparison is the part worth pausing on. The single-device run costs 6.510 h × 1 × $2.50 = $16.28; the eight-device run costs $19.15. You bought a 5.55-hour reduction in wall-clock time for $2.87 — and the extra spend is exactly the 1 ÷ 0.85 = 17.6% efficiency loss applied to the base cost: $16.28 × 1.176 = $19.15.
How to read the result and what scaling efficiency to expect
Read total steps first, because the learning-rate schedule is defined against it. Warm-up is usually specified as a step count or a fraction of total steps, and cosine or linear decay schedules need the total to be known in advance. If you change the device count and leave the schedule alone, the step count changes underneath it and the run decays at the wrong rate.
Read the effective batch second. It is the number that governs optimisation behaviour, and it changes whenever the device count changes even though the per-device batch does not. A recipe tuned at effective batch 256 will not transfer unchanged to effective batch 2,048. The common adjustments — linear or square-root learning-rate scaling with a longer warm-up — are heuristics, not guarantees, and they need validating on your own loss curve.
Scaling efficiency is the figure to be sceptical about. Within a single node connected by NVLink, well-tuned data parallelism on a compute-heavy model commonly holds above 90%. Across nodes over Ethernet, on a small model where the gradient exchange is large relative to the arithmetic, it can fall well below 70%. The only reliable way to know is to measure: run the same job on 1, 2 and 4 devices, record the achieved step rate, and divide. The efficiency table above lets you see how much the answer moves before you commit.
Finally, treat the hours figure as a lower bound on calendar time. It excludes dataset loading and shuffling on the first epoch, checkpoint writes, mid-training evaluation, queue waits and node failures. Long runs need resumable checkpoints; the warning above fires past two weeks for exactly that reason.
Steps per epoch by dataset size and effective batch
| Samples | Effective batch 32 | Effective batch 128 | Effective batch 512 | Effective batch 2,048 |
|---|---|---|---|---|
| 50,000 | 1,563 | 391 | 98 | 25 |
| 200,000 | 6,250 | 1,563 | 391 | 98 |
| 1,000,000 | 31,250 | 7,813 | 1,954 | 489 |
| 10,000,000 | 312,500 | 78,125 | 19,532 | 4,883 |
Effective batch is per-device batch times device count. The ceiling matters most in the right-hand columns, where the partial final batch is a larger share of an epoch.
What makes a training-time estimate wrong
- Measuring the step rate at a different batch size. Steps per second falls as the micro-batch grows. A rate measured at batch 8 applied to a job at batch 64 understates the time by close to the batch ratio.
- Timing the first ten steps. Compilation, kernel autotuning, memory-allocator warm-up and the first dataset shuffle all land in the opening seconds. Discard them before computing a rate.
- Assuming scaling efficiency holds as the cluster grows. Gradient exchange grows with device count while per-device arithmetic does not, so the efficiency you measured on 8 devices is an upper bound for 64.
- Forgetting gradient accumulation. If you accumulate over 4 micro-batches before stepping, the effective batch is 4 times larger and the step rate is roughly 4 times lower. Enter the accumulated batch, not the micro-batch.
- Leaving evaluation out. A validation pass every 500 steps on a large held-out set can add 10% or more to the calendar; the exact figure is your eval-set size divided by inference throughput, times the number of passes.
- Ignoring the data pipeline. If the loader cannot keep 8 devices fed, the achieved step rate is set by disk and CPU, not by the accelerator, and adding devices changes nothing at all.
When to use a different estimate instead
This calculator works from a measured step rate, which makes it the right tool once you have a job running or a comparable one to benchmark. When you have neither — you are sizing a pre-training run that does not exist yet — work from arithmetic instead: the LLM training compute calculator applies C = 6ND to get total FLOPs and divides by sustained hardware throughput, which needs no step-rate measurement at all. The two agree when both are calibrated honestly, and disagreeing is a useful signal that one of your assumptions is wrong.
For model-parallel and pipeline-parallel jobs, the model here breaks down. Those strategies split a single model across devices rather than replicating it, so adding devices does not increase the effective batch and the bottleneck becomes pipeline bubbles and activation transfer rather than gradient all-reduce. Most large runs combine all three forms of parallelism, in which case the scaling efficiency you enter should be measured end to end on the real topology.
Before you can run anything you have to fit it in memory, which is a separate constraint entirely: the GPU VRAM requirement calculator sizes weights, gradients, optimiser states and activations, and the quantisation memory savings calculator shows what lower precision buys. Once you have both the hours from this page and a device count, the GPU-hours cost calculator turns them into an invoice across instance types and commitment terms.