The trade fine-tuning actually offers
Both paths get the model to behave the way you want. The long-prompt path pays for that behaviour on every single request, in tokens: a block of few-shot examples and extended instructions that is re-sent, re-read and re-billed each time. The fine-tuning path pays for it once, in a training run, and then serves shorter requests forever.
That is a classic fixed-versus-variable trade, so it has a break-even and the break-even is measured in requests. Below it the long prompt is cheaper because the training bill has not been amortised. Above it the fine-tune is cheaper because each request costs less.
What makes the comparison less obvious than it looks is a second, opposing term. Hosted fine-tuned models frequently carry a higher per-token rate than the base model they came from, and that premium applies to every token you still send — the user's message, the retrieved context and the whole reply. So fine-tuning does not simply remove cost; it removes the prompt block and adds a surcharge on everything else.
The per-request difference is therefore a subtraction of two effects:
δ = (tokens removed × base rate − tokens remaining × rate premium) ÷ 1,000,000
When δ is positive, fine-tuning is cheaper per request and break-even exists. When δ is zero or negative, no volume repays the training cost, and any case for fine-tuning has to be made on quality or latency instead. That sign check is the first thing to run, and it is the thing spreadsheets usually miss because they compare the prompt saving against zero rather than against the premium.
Working the formula through
Write the cumulative cost of each path over N requests. Let t₀ be the tokens both paths send, t_e the extra prompt tokens only the prompting path sends, b the base rate and f the fine-tuned rate, both per million tokens.
Prompting: N × (t₀ + t_e) × b ÷ 10⁶
Fine-tuned: T + N × t₀ × f ÷ 10⁶
Set them equal and solve for N:
N* = T ÷ [(t_e·b − t₀(f − b)) ÷ 10⁶]
The bracket is δ, the per-request difference, and it decomposes cleanly. The first term, t_e·b, is the saving from tokens you no longer send. The second, t₀(f − b), is the premium on the tokens you still send. Break-even is the training cost divided by their difference.
Three implications follow.
A long request makes fine-tuning worse, not better. Raising t₀ — longer replies, more retrieved context — raises the premium term while leaving the saving term untouched. Teams often assume that a token-heavy workload strengthens the case for fine-tuning; when the fine-tuned rate carries a premium, it weakens it.
When the two rates are equal, the premium term vanishes and break-even is simply T ÷ (t_e·b ÷ 10⁶) — training cost divided by the value of the prompt tokens you removed. This is the case for models you host yourself, where a LoRA adapter serves at the same cost as the base model.
Break-even scales linearly with training cost. Doubling the epochs doubles the requests you need. A cheap adapter train on a few thousand examples has a break-even most production endpoints clear in days; a full-parameter fine-tune of a large model may never clear it.
To estimate the training side from first principles rather than from a vendor quote, use the training compute calculator and the GPU hours cost calculator.
Worked example: dropping a 4,000-token few-shot block
Your classifier prompt carries 4,000 tokens of few-shot examples. The rest of the request — the document to classify and the short answer — is about 1,000 tokens. The base model blends to $3.00 per million; the hosted fine-tune of it blends to $6.00. A fine-tuning run on your labelled set costs $300. You serve 200,000 requests a month.
- Cost per request with the long prompt. (1,000 + 4,000) = 5,000 tokens; 5,000 × $3.00 ÷ 1,000,000 = $0.015.
- Cost per request fine-tuned. 1,000 × $6.00 ÷ 1,000,000 = $0.006.
- Difference per request. $0.015 − $0.006 = $0.009. Decomposed: you save 4,000 × $3 ÷ 10⁶ = $0.012 on dropped tokens and pay 1,000 × $3 ÷ 10⁶ = $0.003 of premium on the rest.
- Break-even. $300 ÷ $0.009 = 33,333 requests.
- Monthly cost, long prompt. 200,000 × $0.015 = $3,000.
- Monthly cost, fine-tuned serving. 200,000 × $0.006 = $1,200.
- Months to recover. $300 ÷ $1,800 per month = 0.17 months, about five days.
- Net over twelve months. 12 × $1,800 − $300 = $21,300.
At this volume the decision is easy. Now cut the volume to 5,000 requests a month: the monthly difference is 5,000 × $0.009 = $45, and recovering $300 takes 6.7 months. The same fine-tune, the same rates, an entirely different decision — which is why volume is the input that matters most.
Now add prompt caching to the comparison. If the 4,000-token block is cached and read at a 0.1× multiplier, its effective rate is $0.30 per million, so the saving term falls to 4,000 × $0.30 ÷ 10⁶ = $0.0012 while the premium term stays at $0.003. δ becomes $0.0012 − $0.003 = −$0.0018: the cached long prompt is now the cheaper path at every volume, and fine-tuning never repays its training cost on price. Work out your own cached rate with the prompt caching savings calculator before you commit to training.
How to read the result
Check the sign of the per-request difference before anything else. If it is negative the calculator reports no break-even, and that is the correct answer rather than a missing number: at those rates the fine-tuned path costs more on every request, so more volume makes it worse. The lever is either a smaller serving premium — self-hosting an adapter, or a provider that does not charge one — or a larger prompt block to remove.
Compare break-even against a horizon, not against your total traffic. Anything you clear within a month or two is straightforward. A break-even measured in years is a bet that the model, the prices and the task all stay still for years, and none of them has. The calculator warns when break-even exceeds twelve months of your current volume.
Months to recover is the number to put in front of a budget owner. It converts a request count into a time, and it makes the volume sensitivity obvious: halving traffic doubles the payback period exactly.
Remember what the training cost excludes. The figure you enter should cover the runs you actually make, not the one you hope to make first time — data preparation, a failed hyperparameter choice, evaluation and at least one retrain are normal. And fine-tuning is not a one-off in practice: when the base model is deprecated or your task drifts, you train again. A fine-tune with a six-month payback and a three-month useful life does not pay back.
The quality axis is not in this calculation and often outranks it. Fine-tuning can produce behaviour that no prompt reliably elicits, particularly for strict output formats, domain vocabulary and consistent tone. It also shortens prompts, which cuts latency — see the throughput and latency calculator for how prompt length maps to time to first token. When either matters, cost is a constraint rather than the decision.
Break-even request count by training cost and per-request difference
| Training cost | $0.0015 / request | $0.003 / request | $0.009 / request | $0.02 / request |
|---|---|---|---|---|
| $100 | 66,667 | 33,333 | 11,111 | 5,000 |
| $300 | 200,000 | 100,000 | 33,333 | 15,000 |
| $1,000 | 666,667 | 333,333 | 111,111 | 50,000 |
| $5,000 | 3,333,333 | 1,666,667 | 555,556 | 250,000 |
For scale, $0.009 per request is the worked example: a 4,000-token block dropped at $3 per million, less a $3 premium on 1,000 remaining tokens. A per-request difference of $0.0015 is what a modest 500-token block at $3 per million yields with no serving premium.
The serving premium applies to the reply, not just the prompt
It is easy to model the fine-tuned rate as applying only to input tokens, because the prompt is what fine-tuning changes. It is not: the premium applies to every billed token, and on a generation-heavy workload the reply may be most of them. A summarisation endpoint that sends 500 tokens and returns 2,000 pays the premium on 2,500 tokens while saving only on the few-shot block it dropped. Put the full input-plus-output count in the tokens-both-paths-send field, and blend the rates over your real input-to-output mix — the token cost calculator gives that breakdown.
What this comparison leaves out
- Data preparation. Assembling, cleaning and labelling training examples is usually the largest real cost of a fine-tune, and none of it appears on a GPU bill.
- Evaluation. You need a held-out set and a way to score it to know the fine-tune is better rather than merely different. Build that before training, not after.
- Retraining. Base models are deprecated and tasks drift. Amortise the training cost over the fine-tune's realistic life, not over an unlimited horizon.
- Hosting minimums. Some providers charge to keep a fine-tuned model deployed regardless of traffic, which is a fixed monthly cost this model does not include. Add it to the training cost divided by the months you expect to run.
- Quality differences in both directions. A fine-tune can be better at the task and worse at everything else. If one endpoint handles several jobs, a narrow fine-tune may need a router in front of it.
- Prompt caching on the other side. A cached few-shot block is far cheaper than an uncached one, which moves break-even sharply. Compare against the cached alternative, not the naive one.
The order to try things in
Fine-tuning is rarely the first thing to reach for, because three cheaper interventions attack the same cost and take hours rather than days.
Prune the prompt. Few-shot blocks accumulate. Measure how many examples actually change the output — often a fraction of what is there — and delete the rest. This costs nothing and reduces the very term fine-tuning is supposed to remove.
Cache what remains. A static block read at a fraction of the base rate is close to free, and as the worked example shows it can reverse the fine-tuning decision outright. Caching is a configuration change; fine-tuning is a project.
Try a smaller model with the long prompt. A cheaper base model carrying the full few-shot block sometimes beats a fine-tuned larger one on both cost and quality, and it costs one afternoon to test.
After those, fine-tuning earns its place in three situations: when the behaviour cannot be prompted reliably at any prompt length; when latency matters and the prompt is long enough that prefill is the bottleneck; and when volume is high enough that even a modest per-request difference clears the training cost quickly. The first two are quality and latency arguments that this calculator does not price. The third is exactly what it prices.
Finally, put the answer in business terms. A monthly saving is an infrastructure number; the question a pricing decision turns on is cost per user and gross margin. Feed the reduced per-request cost into the cost per user per month calculator to see whether the fine-tune moves the line that actually matters.