What chunking decides in a retrieval pipeline
Chunking is the step where you cut documents into the units that get embedded, indexed and retrieved. It is the highest-leverage decision in a retrieval-augmented generation pipeline, because chunk size simultaneously sets three things that pull against each other: how precisely a match can be localised, how much surrounding context arrives with it, and how much of the model's window each retrieved item consumes.
Small chunks give sharp retrieval. A 128-token chunk that matches a query is almost entirely about the query, so the embedding is not diluted by unrelated text. But a small chunk often lacks the context needed to answer — the paragraph that defines the term, the table header two paragraphs up — and you need more of them, which multiplies index size and query cost.
Large chunks carry their context with them and there are fewer to store, but their embeddings average over several topics, so a chunk about six things is a weak match for a query about one of them. And each retrieved chunk is expensive: at top-K = 5, moving from 512-token chunks to 2,048-token chunks takes retrieved context from 2,560 tokens to 10,240, which changes both what fits in the window and what every query costs.
Overlap sits underneath all of this. Fixed-size splitting cuts at arbitrary token boundaries, which will sooner or later cut a sentence, a definition or a table row in half. Repeating the last o tokens of each chunk at the start of the next means the split content survives intact in at least one chunk. You pay for it in duplicated storage, and this calculator reports that multiple directly.
Why the formula uses stride rather than chunk size
The count is governed by the stride, c − o, not by the chunk size. A sliding window of width c that advances c − o tokens at a time places its first chunk at position 0 and each subsequent one o tokens earlier than a naive split would. So the number of windows needed to reach the end of an L-token document is ceil((L − o) ÷ (c − o)).
Check it at the boundaries. When L = c, the expression gives (c − o) ÷ (c − o) = 1, one chunk, correct. When L = c + stride, it gives 2, correct. When o = 0 the stride equals the chunk size and the formula reduces to the familiar ceil(L ÷ c). And when L < c it returns a value below 1, which is why the result is clamped upward: a 200-token document is still one chunk.
The failure mode is o ≥ c. Then the stride is zero or negative, the window never advances, and the chunk count is not merely large — it is undefined. This calculator returns a dash rather than infinity for that case, because an infinite value in a pipeline configuration is a bug, not a number.
Storage amplification follows from the same geometry. All but the last chunk store a full c tokens while advancing only c − o, so in the limit of a long document each source token appears in c ÷ (c − o) chunks. At c = 512 and o = 64 that is 512 ÷ 448 = 1.143×; at 50% overlap it is exactly 2×. Short documents amplify less, because the final partial chunk is not padded.
Worked example: 5,000 documents of 1,200 tokens at chunk 512, overlap 64
You have 5,000 support articles averaging 1,200 tokens each. You split at 512 tokens with 64 tokens of overlap, retrieve the top 5 chunks, and send them to a model with an 8,192-token window alongside a 500-token system prompt, reserving 1,000 tokens for the answer.
- Stride. 512 − 64 = 448 tokens between chunk starts.
- Chunks per document. (1,200 − 64) ÷ 448 = 1,136 ÷ 448 = 2.536, and ceil(2.536) = 3 chunks. Their spans are 0–512, 448–960 and 896–1,200.
- Total chunks. 3 × 5,000 = 15,000 chunks, which is the vector count for your index.
- Storage amplification. The first two chunks store 512 tokens each; the third stores 1,200 − 896 = 304. Total stored = 1,024 + 304 = 1,328 against 1,200 source tokens, so 1.107×.
- Retrieved context. 5 × 512 = 2,560 tokens.
- Total request. 2,560 + 500 + 1,000 = 4,060 tokens.
- Window used. 4,060 ÷ 8,192 = 49.6%, leaving 4,132 tokens of headroom.
Now test the same corpus at 2,048-token chunks with the same 12.5% overlap ratio, which is 256 tokens. Stride is 1,792, so ceil((1,200 − 256) ÷ 1,792) = ceil(0.527) = 1 chunk per document and 5,000 chunks total — a third of the index. But retrieved context becomes 5 × 2,048 = 10,240 tokens, and 10,240 + 500 + 1,000 = 11,740 against an 8,192 window. The configuration that saves two thirds of the storage does not fit at all.
How to read the result and choose a chunk size
Check headroom before anything else. A negative headroom is a hard failure: the request will be rejected or silently truncated, and truncation usually removes the end of the context, which is where your last-ranked and often most specific chunk sits. Leave real slack, because your average document length is not your longest and a multi-turn conversation grows the prompt on every turn.
Then look at whether the chunk size matches what a good answer needs. There is no universal optimum, and published comparisons disagree because the answer depends on your documents. A practical rule: the chunk should be large enough to contain a complete answerable unit — a full procedure, a full definition, a full table with its header — and no larger. For prose documentation that is often 256–512 tokens; for legal or technical reference material with long self-contained clauses, 1,000 or more.
Read total chunks as a cost driver, since it flows directly into two other budgets. It is the vector count for the vector database storage calculator, and stored tokens (source tokens times the amplification figure) is the input to the embedding cost calculator. Halving the chunk size roughly doubles both.
Set overlap as a fraction of the chunk rather than an absolute number. Something in the 10–20% range keeps boundary-crossing content intact without a punishing storage multiple; at 50% you are storing every source token twice, paying twice to embed it, and putting two near-identical chunks into competition for the same top-K slots. If your retrieval results routinely contain two chunks that are mostly the same text, high overlap is why.
Chunks per document by document length and chunk size
| Document length | c = 256, o = 25 | c = 512, o = 50 | c = 1,024, o = 100 |
|---|---|---|---|
| 500 tokens | 3 | 1 | 1 |
| 2,000 tokens | 9 | 5 | 3 |
| 10,000 tokens | 44 | 22 | 11 |
Strides are 231, 462 and 924 tokens respectively. Note that halving the chunk size a little more than doubles the count, because the overlap scales with it.
Chunking mistakes that show up as bad retrieval
- Counting characters and calling them tokens. English averages roughly 4 characters per token, so a 2,000-character splitter produces about 500-token chunks. Configure the splitter in the same unit your embedding model's limit is expressed in.
- Exceeding the embedding model's input limit. Most providers truncate silently rather than erroring. A 4,000-token chunk sent to a 512-token model is embedded from its first 512 tokens, and the rest of the chunk is unreachable by search while still being returned in full to the LLM.
- Splitting tables and code blocks mid-structure. A table row without its header is close to meaningless both to the embedder and to the model. Structure-aware splitting on headings, then fixed-size splitting only within an oversized section, avoids most of this.
- Forgetting that top-K multiplies chunk size. The context cost of a chunking decision is K × c, not c. Raising K from 5 to 20 at 512 tokens costs the same context as raising the chunk to 2,048 at K = 5, but produces very different retrieval behaviour.
- Reserving no space for the answer. On APIs where input and output share one window, the maximum output tokens must be subtracted before you decide how much context fits.
- Changing chunk size without re-indexing. Chunk boundaries are baked into the stored vectors. A new chunk size means re-embedding the whole corpus, which is the cost most teams discover after the fact.
Fixed-size chunking and the alternatives to it
What this calculator models is fixed-size sliding-window chunking, the default in every mainstream framework and the only strategy whose chunk count is predictable from arithmetic alone. Three common alternatives change the count in ways you have to measure rather than derive.
Recursive or structure-aware splitting cuts on the largest natural boundary that fits — heading, then paragraph, then sentence — and falls back to fixed size only inside an oversized block. Chunk counts land close to the fixed-size estimate but individual chunks are shorter, because a split lands early at a paragraph break rather than exactly at the limit. Treat this calculator's output as an upper bound on chunk size and a lower bound on chunk count.
Semantic chunking splits where consecutive sentence embeddings diverge, producing variable-length chunks aligned to topic shifts. The count depends entirely on the document and cannot be predicted in advance; run it on a sample and measure.
Hierarchical or parent-document retrieval decouples the two competing pressures: embed small chunks for precise matching, then return the larger parent section to the model. That changes the arithmetic on this page in an important way — retrieved context becomes K times the parent size, not K times the indexed chunk size, so recompute your window budget against the parent. It is the standard answer when small chunks retrieve well but answer badly.
Whatever you choose, the downstream numbers stay the same. Chunk count sizes the index, stored tokens price the embedding pass, and retrieved context per query drives both latency and the per-request cost you can work out with the LLM API token cost calculator or, if the same system prompt precedes every request, the prompt cache savings calculator.