Why generation speed is a bandwidth problem
To produce one output token, a transformer reads every weight in the model exactly once. Each decode step multiplies a single new token's activations against the full weight matrices, so the arithmetic per weight is tiny — roughly two floating-point operations — while the data movement per weight is its full size in bytes. Modern accelerators can do hundreds of arithmetic operations in the time it takes to fetch one byte from high-bandwidth memory, so the fetch is what you wait for.
That gives a startlingly simple ceiling. If the weights occupy S gigabytes and memory delivers B gigabytes per second, you cannot generate faster than B ÷ S tokens per second per stream. A 70-billion-parameter model at FP16 occupies 140 GB; on 3,350 GB/s of HBM3 that is 23.9 tokens per second, and no amount of extra compute changes it.
Two consequences follow immediately, and both are counter-intuitive the first time you meet them. Halving the precision roughly doubles the speed, because you halve the bytes fetched per weight — this is why quantisation is a latency technique as much as a memory technique. And batching is nearly free for the individual user: one weight read serves every stream in the step, so eight concurrent users decode at roughly the same per-stream speed as one, while aggregate throughput multiplies by eight.
Prompt processing behaves in the opposite way. Prefill runs all the prompt tokens through the model in parallel, so it does hundreds of arithmetic operations per fetched byte and is limited by compute rather than by memory. That is why the first token can take a fraction of a second after a 2,000-token prompt while each subsequent token takes tens of milliseconds.
The formula and what each term hides
v = B × η ÷ S is the whole decode model. The details live in the three terms.
S, model size in memory. Parameters times bytes per weight, not the size of the checkpoint on disk if the two differ. A 7B model is 14 GB at FP16, 7 GB at INT8 and about 3.5 GB at INT4. If the model is split across several GPUs with tensor parallelism, use the total size and the total bandwidth — each GPU holds a shard and reads it concurrently, so the ratio is what matters. The quantisation memory savings calculator gives the footprint at each precision.
B, memory bandwidth. Take it from the GPU datasheet and add it up across the GPUs holding the weights. Bandwidth, not FLOPS, is the specification that predicts decode speed — a card with twice the tensor throughput and the same memory system generates text at the same rate.
η, achieved fraction of peak. No kernel attains datasheet bandwidth. Memory access patterns, kernel launch overheads, tensor-parallel communication and imperfect overlap all cost you some of it. Use a measured number from your own stack if you have one, and treat the result at η = 100% as a ceiling rather than a prediction.
The two latency terms are simple division. Time to first token is prompt tokens divided by the prefill rate. Generation time is output tokens divided by v. Total response time is their sum, and inter-token latency — the interval between characters appearing on screen — is 1,000 ÷ v milliseconds.
One term is deliberately absent: the KV cache. Every decode step also reads the cached keys and values for the whole sequence so far, and that read grows with context length and batch size. At short context it is small next to the weights; at long context it is not, which is why measured speed falls as a conversation gets longer. Size it with the KV cache size calculator and treat this page's figure as the ceiling you approach at short context.
Worked example: a 70B model at FP16 on one H100-class GPU pair
You serve a 70-billion-parameter model at FP16, so the weights occupy 70 × 2 = 140 GB. The GPUs supply 3,350 GB/s of aggregate memory bandwidth and your serving stack achieves 85% of it. You batch 8 concurrent requests. A typical request carries 2,000 prompt tokens and generates 500 output tokens, and you measure prefill at 9,000 tokens per second.
- Usable bandwidth. 3,350 × 0.85 = 2,847.5 GB/s.
- Per-stream rate. 2,847.5 ÷ 140 = 20.34 tokens per second. Each token appears 1,000 ÷ 20.34 = 49.2 ms after the last.
- Aggregate throughput. 20.34 × 8 = 162.7 tokens per second across the batch.
- Time to first token. 2,000 ÷ 9,000 = 0.222 s.
- Generation time. 500 ÷ 20.34 = 24.58 s.
- Total response time. 0.222 + 24.58 = 24.80 s.
- Requests per hour. 8 streams × 3,600 s ÷ 24.80 s = 1,161 requests.
Now quantise to INT8. The weights fall to 70 GB, so the per-stream rate becomes 2,847.5 ÷ 70 = 40.68 tokens per second — exactly double, because the divisor halved. Generation time drops to 500 ÷ 40.68 = 12.29 s and the response completes in 12.51 s instead of 24.80 s. The prompt phase did not change, because prefill was never bandwidth-limited.
That is the single most useful lesson in the model: on decode, memory footprint is latency. Anything that shrinks the bytes read per token — lower precision, a smaller model, weight sparsity — speeds up generation in direct proportion.
How to read the result
Per-stream tokens per second is what the user experiences; aggregate is what you pay for. They answer different questions and improve under different actions. Raising batch size lifts aggregate throughput and leaves per-stream speed roughly alone, which is exactly what you want for a busy service. Shrinking the model lifts both.
For a rough sense of scale on the per-stream figure: comfortable adult reading is a few words per second, and a token is usually a fraction of a word, so a stream in the tens of tokens per second produces text at or above reading pace. Below about ten tokens per second the text arrives visibly slower than a person reads, which is the threshold the calculator flags.
Time to first token and inter-token latency are separate problems with separate fixes. A long silence before the first character is a prefill problem, and the levers are a shorter prompt, a cached prefix, or more compute. Slow text after it starts is a decode problem, and the levers are a smaller model, a lower precision, or more bandwidth. Applying the decode fix to a prefill complaint is a common and expensive mistake. If your prompt is long and mostly unchanging, the prompt caching savings calculator shows what skipping that prefill is worth in money as well as in seconds.
Requests per hour is the number that connects to capacity planning. It assumes every stream in the batch is busy for the whole response, which is the optimistic case; real schedulers see streams finish at different times and refill the batch continuously. Feed the aggregate throughput figure into the self-hosted versus API break-even calculator to convert speed into monthly capacity and cost per million tokens.
Treat every figure here as an upper bound. The model ignores KV cache reads, scheduling gaps, tokenisation, network time and any queueing before the request reaches a GPU. Measured throughput below this number is normal; measured throughput above it means one of your inputs is wrong.
Theoretical per-stream tokens per second by model size and bandwidth
| Weights in memory | 1,008 GB/s (RTX 4090) | 2,039 GB/s (A100 80GB SXM) | 3,350 GB/s (H100 SXM) |
|---|---|---|---|
| 14 GB (7B at FP16) | 72.0 | 145.6 | 239.3 |
| 35 GB (70B at INT4) | 28.8 | 58.3 | 95.7 |
| 70 GB (70B at INT8) | 14.4 | 29.1 | 47.9 |
| 140 GB (70B at FP16) | 7.2 | 14.6 | 23.9 |
The table ignores whether the model fits: 140 GB does not fit on a single 24 GB or 80 GB card, and serving it needs several GPUs whose bandwidths add up. Read the rows as ratios rather than as configurations.
Batching is free for the user only up to a point
The claim that per-stream speed is unchanged by batching holds while the decode step is memory-bound. Each step reads the weights once and does arithmetic proportional to the batch size, so there is a batch size at which the arithmetic takes longer than the fetch and the step stops being free. Beyond it, aggregate throughput keeps rising but more slowly, and per-stream speed starts to fall. Where that crossover sits depends on the accelerator's ratio of compute to bandwidth and on how much KV cache each sequence carries. The calculator warns above batch 32 because that is where the linear model starts to overstate reality for typical hardware — but the exact point is something you measure, not something you look up.
What this model leaves out
- KV cache reads. Each step also reads the cached keys and values for every token so far. At 8k context on a large batch this can rival the weight read, and it is why speed decays as a conversation lengthens.
- Tensor-parallel communication. Splitting a model across GPUs adds an all-reduce per layer. It is real overhead and shows up as a lower achieved efficiency rather than as a separate term.
- Scheduler behaviour. Continuous batching admits and retires requests mid-flight, so the effective batch size varies second by second. The fixed batch here is a simplification.
- Speculative decoding. Drafting several tokens with a small model and verifying them in one pass with the large one breaks the one-read-per-token assumption and can exceed the ceiling shown here.
- Queueing. Under load, waiting for a slot dominates time to first token. That is a capacity problem, not a speed problem, and no bandwidth figure predicts it.
- Mixture-of-experts models. Only the active experts are read per token, so the effective S is well below the full parameter count. Use active parameters, not total, for those architectures.
Using this alongside memory and cost sizing
Speed, memory and money are three views of the same deployment, and they constrain each other. Memory decides whether the model runs at all and how many sequences fit; the GPU VRAM requirement calculator answers that first question, and the KV cache calculator answers how much room is left for concurrency after the weights are loaded. Concurrency then feeds straight back into this page as batch size, which sets aggregate throughput.
Cost follows from throughput. Aggregate tokens per second times the seconds in a month times your realistic utilisation gives monthly capacity, and dividing the hardware bill by that capacity gives a cost per million tokens you can put next to an API rate. That chain is what the break-even calculator automates, and it is why an efficiency assumption on this page propagates all the way to a hosting decision.
If the numbers here are disappointing, the ordered menu of fixes is short and each item is quantified elsewhere on the site. Quantise, which divides S and multiplies speed by the same factor. Shorten the context, which shrinks the KV cache term this model ignores and speeds up prefill at the same time. Raise batch size, which multiplies aggregate throughput at no per-stream cost until the arithmetic limit bites. Only then buy more bandwidth — it is the most expensive lever and the last one to reach for.
Key terms
- Prefill
- The first phase of a request, where all prompt tokens are processed in parallel to build the KV cache. Compute-bound, and fast per token.
- Decode
- The generation phase, producing one token at a time. Memory-bound, because each step reads the whole model regardless of how few tokens it produces.
- Time to first token (TTFT)
- The delay between sending a request and the first output token arriving. Dominated by prefill and by any queueing before the request starts.
- Inter-token latency
- The interval between consecutive output tokens, equal to 1,000 ÷ tokens per second in milliseconds. This is what a user perceives as typing speed.
- Arithmetic intensity
- Floating-point operations performed per byte fetched from memory. Decode has very low arithmetic intensity, which is precisely why bandwidth bounds it.