What a GPU-hour is and why it is the unit that matters
Cloud accelerators are billed per GPU per hour, so every training run, fine-tune and batch inference job reduces to one product: rate × GPUs × hours. Holding eight GPUs for seventy-two hours is 576 GPU-hours whether the job used them well or left them idle, because the meter runs on allocation rather than on utilisation.
That single fact drives most of the waste in machine-learning budgets. A job that spends 30% of its wall-clock time waiting on a data loader still pays for 100% of the GPU-hours. A cluster left up over a weekend because nobody wrote the teardown step costs the same as one training productively. GPU-hours are unforgiving in a way that per-request API pricing is not, and the discipline they demand is different: you are managing allocated time, not consumed work.
The second thing this calculator prices is the interruptible tier — spot on AWS, preemptible or Spot VMs elsewhere — where the provider sells you spare capacity at a steep discount and may reclaim it with little notice. The discount is real and large. The catch is that a reclaimed instance loses whatever work has happened since the last checkpoint, and that lost work has to be redone on hours you also pay for.
Netting those two effects correctly is the point of the calculation, and it is where intuition fails: people subtract the overhead from the discount, when in fact the discount multiplies the price and the overhead multiplies the hours.
Why the discount and the overhead multiply rather than subtract
The on-demand cost is a plain product:
C = r × g × t
Spot changes two different terms. The discount lowers r to r(1 − d). The interruptions raise t to t(1 + o), because the work redone after each preemption occupies GPUs that are metered exactly like the rest. So:
C_spot = r(1 − d) × g × t(1 + o) = C × (1 − d)(1 + o)
The cost factor (1 − d)(1 + o) is the whole comparison. Below 1.0 spot is cheaper; above 1.0 it is dearer. Set it equal to 1 and solve for the overhead at which spot stops paying:
(1 − d)(1 + o) = 1 → o* = d ÷ (1 − d)
This threshold is much higher than most people guess. At a 50% discount you can afford 100% overhead — redoing the entire job — before spot loses. At 65% you can afford 186%. At 80% you can afford 400%. Interruptible capacity is remarkably robust to interruptions, provided you can actually resume.
The word doing the work in that sentence is resume. The formula assumes lost work is bounded by the checkpoint interval. A job that cannot restart from a checkpoint does not have an overhead of 200%; it has an overhead that grows without limit, because each preemption throws away everything. The model is a cost model for a job engineered to survive preemption, not an argument that any job can use spot.
Estimating o. With preemptions arriving at a mean interval and checkpoints written every c hours, the work lost per preemption averages roughly half a checkpoint interval, so the overhead is about c ÷ 2 divided by the mean uninterrupted run length, plus the time to write each checkpoint and reload state. If you have no history to go on, measure a short run before committing a long one.
Worked example: a 72-hour fine-tune on eight GPUs
You are fine-tuning on eight GPUs at $2.50 per GPU-hour on demand. The job takes 72 hours of wall clock. Interruptible capacity in your region is offered at a 65% discount, and from a previous run you know that checkpointing every 30 minutes and resuming automatically costs about 15% extra GPU-hours.
- GPU-hours. 8 × 72 = 576 GPU-hours.
- On-demand cost. 576 × $2.50 = $1,440.00.
- Cost factor. (1 − 0.65) × (1 + 0.15) = 0.35 × 1.15 = 0.4025.
- Spot cost. $1,440.00 × 0.4025 = $579.60.
- Saving. $1,440.00 − $579.60 = $860.40, which is 860.40 ÷ 1,440 = 59.75% of the on-demand bill.
- Effective rate. $579.60 ÷ 576 GPU-hours = $1.00625 per GPU-hour against a $2.50 list price.
- Break-even overhead. 0.65 ÷ 0.35 = 1.857 = 185.7%. Your 15% is nowhere near it.
Note how far the 59.75% saving sits below the 65% headline discount: the 15% overhead costs you 5.25 percentage points of it, because 0.35 × 1.15 = 0.4025 rather than 0.35. That gap is the part left out of a naive comparison, and it is small here precisely because the discount is large.
For context on the fleet, the same eight GPUs held continuously for a month would cost $2.50 × 8 × 730 = $14,600 on demand. If your fine-tuning cadence is one 72-hour run a week, you are using about four of those runs a month — $5,760 on demand or $2,318 on spot — so keeping a permanent reservation would be the more expensive choice unless it also serves inference.
How to read the result
Look at the cost factor before the dollar figure. It is the scale-free version of the answer and it is the only number that transfers between jobs. A factor of 0.4025 means this job on interruptible capacity costs 40% of list, and that ratio holds whether the run is 72 hours or 720.
Compare your overhead against the break-even overhead, not against zero. The threshold d ÷ (1 − d) rises steeply with the discount, and knowing it changes the engineering decision: at a 65% discount there is no point spending a week making checkpoints twice as frequent to shave the overhead from 15% to 8%, because the saving is 2.5 percentage points of the bill. At a 30% discount, where break-even is only 42.9%, that same work might be what keeps spot viable at all.
The effective rate is the number to quote and to compare. It already carries the restart penalty, so it is directly comparable to another provider's on-demand rate, to a reserved-instance rate, or to the amortised hourly cost of hardware you own.
Treat the monthly continuous figure as the reservation test. If your intermittent jobs add up to a large share of it, a committed-use discount or owned hardware starts to compete. If they add up to a small share, on-demand or spot is almost certainly right, because a reservation you do not fill is a bill you pay for idle silicon.
Once you know the cost of a GPU-hour, the remaining question is how many you need. Estimate that from the model and dataset with the training time estimate calculator and the training compute calculator, then multiply. For serving rather than training, the self-hosted versus API break-even calculator converts the same hourly rate into a cost per million tokens.
Spot cost factor by discount and interruption overhead
| Discount | 0% overhead | 25% | 50% | 100% | Break-even overhead |
|---|---|---|---|---|---|
| 30% | 0.700 | 0.875 | 1.050 | 1.400 | 42.9% |
| 50% | 0.500 | 0.625 | 0.750 | 1.000 | 100.0% |
| 65% | 0.350 | 0.438 | 0.525 | 0.700 | 185.7% |
| 80% | 0.200 | 0.250 | 0.300 | 0.400 | 400.0% |
The break-even column is discount ÷ (1 − discount). At a 30% discount a 50% overhead already costs more than on demand; at 80% you could redo the job four times over and still be ahead.
The meter runs on allocation, not on utilisation
You pay for a GPU from the moment it is allocated to the moment it is released, including the time it spends waiting for data, waiting for a slow all-reduce, waiting for you to notice the job crashed at 2 a.m., and sitting idle after the run finished because nothing tore the cluster down. A job with 60% GPU utilisation costs exactly the same as the same job at 95%. Before optimising the rate, check the utilisation: raising it from 60% to 90% is a 33% cut in hours, which usually beats anything a pricing tier can offer, and it applies to on-demand and spot alike.
What this figure leaves out
- The host instance. GPU instances bundle CPUs, host RAM and local NVMe, and providers usually quote a single instance price rather than a bare GPU price. If your rate is per instance, divide by the GPUs in it before entering it here.
- Storage and data movement. Datasets, checkpoints and logs sit in object or block storage that is billed separately, and checkpointing more often to reduce interruption overhead writes more of it.
- Egress. Moving model artefacts or datasets out of a region is charged per gigabyte and can be a meaningful share of a small job's cost.
- Idle time between stages. A pipeline that holds GPUs during data preparation pays GPU rates for CPU work. Split the stages if the gap is long.
- Queue time on spot. Waiting for interruptible capacity to become available costs no money but does cost calendar time, and a deadline missed has its own price.
- Failed runs. Hyperparameter mistakes, diverged losses and out-of-memory crashes consume real GPU-hours. Budget a realistic number of restarts on top of the successful run.
Choosing between on-demand, spot and a commitment
The three purchasing modes suit three different demand shapes, and the calculator's outputs map onto them directly.
On-demand suits work that must finish by a deadline and cannot tolerate a restart: a demo the day before it is shown, an interactive debugging session, an inference service with a latency commitment. You pay list price for the right to be uninterrupted.
Spot or preemptible suits anything checkpointable and schedule-flexible, which describes most training and nearly all batch inference. The break-even overhead is high enough that the engineering to survive preemption almost always pays for itself. The prerequisites are real, though: frequent checkpoints, automatic resume, and a job that can start on a different node than it stopped on.
Committed use or owned hardware suits steady, predictable, continuous demand. The monthly continuous figure on this page is the yardstick: compare your actual monthly GPU-hours against it, and if you are consistently using a large fraction, a commitment converts a variable bill into a lower fixed one. If you are not, a commitment is a bill for idle time.
One structural note on training specifically. The GPU-hours a training run needs are set by the model size, the dataset size and the arithmetic throughput you achieve, not by the price of a GPU-hour. Estimate the hours first — the training compute calculator gives the total floating-point operations and the training time calculator converts them into wall clock — and only then choose the tier you buy them in. Choosing hardware before knowing the compute requirement is how teams end up paying for a fleet that is either too small to finish or too large to justify.