What perplexity measures
Perplexity is the effective number of equally likely options a language model is choosing between at each token. A perplexity of 12 means the model is about as uncertain as someone picking uniformly from 12 possibilities — not that there are 12 candidates, but that its probability distribution carries the same uncertainty as a fair 12-sided die.
It is defined as the exponential of the cross-entropy loss, and cross-entropy is the mean of −log p over every token in the evaluation set, where p is the probability the model assigned to the token that actually appeared. Assign high probability to what happened and the loss is low; assign low probability and the loss is large. Because the log is taken of a probability, which is at most 1, cross-entropy is never negative, and perplexity is therefore never below 1.
The reason both numbers exist is that they suit different uses. Loss is additive, differentiable and well behaved as a training signal, so that is what optimisers minimise and what training curves plot. Perplexity is interpretable — a branching factor you can reason about — so that is what papers report. Bits per token is the same quantity in base 2, and it is the natural unit when you are thinking about compression: a model with 3.6 bits per token can, in principle, encode text at 3.6 bits per token.
The one fixed reference point is the uniform baseline. A model that assigns equal probability to all V vocabulary entries has perplexity exactly V. That is what "learned nothing" looks like, and it is the number your result should first be compared against.
Why the base cancels, and how nats and bits relate
The conversions look like three formulas and are really one. Cross-entropy in nats uses the natural log; in bits it uses log base 2; and log2(x) = ln(x) ÷ ln 2, so bits = nats ÷ 0.693147, and nats = bits × 0.693147. Perplexity is recovered by exponentiating in the same base the log was taken in: exp(nats) and 2bits give identical numbers, because 2H/ln 2 = eH.
That identity is worth internalising, because reading a loss in the wrong units is the most common mistake with this metric and it produces a plausible-looking wrong answer. A loss of 2.5 is perplexity 12.18 if it is in nats and 5.66 if it is in bits — both are perfectly reasonable numbers for a language model, so nothing about the result flags the error. Deep-learning frameworks compute cross-entropy with the natural log by default, so a training curve is in nats unless you changed it.
Two anchors make the scale concrete. A loss of ln 2 = 0.6931 nats is exactly 1 bit and exactly perplexity 2: the model is as uncertain as a fair coin. A loss of ln V is exactly the uniform baseline: log(50257) = 10.825 nats, 15.617 bits, perplexity 50,257.
Word-level perplexity is a different exponent on the same loss. Token perplexity is exp of loss per token; word perplexity is exp of loss per word, and loss per word is loss per token times tokens per word. So PPLword = PPLtokenr, where r is your tokeniser's tokens-per-word ratio. This conversion is what makes a modern subword model comparable with the classic word-level perplexities reported on benchmarks such as WikiText-103, and getting r wrong distorts the comparison exponentially.
Worked example: a validation loss of 2.5 nats on a 50,257-token vocabulary
Your run reports a validation cross-entropy of 2.50 nats. The tokeniser has a 50,257-entry vocabulary and averages 1.3 tokens per word on this corpus.
- Perplexity. exp(2.50) = 12.1825. The model is as uncertain per token as a uniform choice among about 12 options.
- Bits per token. 2.50 ÷ ln 2 = 2.50 ÷ 0.693147 = 3.6067 bits. Check it the other way: 23.6067 = 12.1825, the same perplexity.
- Uniform baseline. A model that guessed uniformly would have perplexity 50,257, which is log2(50,257) = 15.6169 bits per token.
- Improvement factor. 50,257 ÷ 12.1825 = 4,125× lower perplexity than uniform guessing.
- Bits saved. 15.6169 − 3.6067 = 12.0102 bits per token. On a 1,000-token document that is 12,010 bits, or about 1.5 kilobytes of compression relative to a naive fixed-width encoding of the vocabulary.
- Word-level perplexity. Loss per word is 2.50 × 1.3 = 3.25 nats, so word perplexity is exp(3.25) = 25.79. Equivalently, 12.18251.3 = 25.79.
Notice how much larger the word figure is: the same model, the same corpus, and a number more than twice as high, purely because a word carries 1.3 tokens' worth of uncertainty. Any comparison of published perplexities has to establish which unit is being counted before the numbers mean anything.
How to read a perplexity, and what it cannot tell you
Perplexity has no absolute good value, and the honest thing to say about a number in isolation is that it is uninterpretable. It depends on the corpus — predicting boilerplate legal text is far easier than predicting conversational speech — and it depends on the tokeniser, because a larger vocabulary packs more text into each token and mechanically lowers the per-token loss without any improvement in modelling. Two models are comparable on perplexity only when they share both.
What perplexity does support is comparison within a fixed setup, and there it is excellent. On the same held-out set with the same tokeniser, a lower perplexity is a strictly better fit to the data distribution, and the metric is continuous, cheap and low-variance, which is why it is the workhorse of pre-training evaluation. Watching validation perplexity stop falling while training perplexity keeps falling is the classic overfitting signal.
Compare against the uniform baseline to sanity-check the setup rather than to judge quality. A perplexity above the vocabulary size means the model is doing worse than uniform guessing, which on a converged checkpoint almost always indicates a bug: a tokeniser mismatch, a corrupted checkpoint, or evaluation on the wrong data. The calculator warns when that happens.
Two things perplexity cannot measure. It says nothing about whether generated text is useful, truthful or safe — those need task evaluations, and for anything that reduces to a binary decision the confusion matrix metrics calculator and the F1 score calculator are the relevant tools. And it says nothing about behaviour after instruction tuning: RLHF and preference optimisation typically raise perplexity on generic corpora while making the model far more useful, because they deliberately shift the output distribution away from the pre-training distribution.
Cross-entropy loss, bits and perplexity across the usual range
| Loss (nats) | Bits per token | Perplexity | Word-level at 1.3 tokens/word |
|---|---|---|---|
| 0.50 | 0.7213 | 1.6487 | 1.916 |
| 1.00 | 1.4427 | 2.7183 | 3.669 |
| 1.50 | 2.1640 | 4.4817 | 7.029 |
| 2.00 | 2.8854 | 7.3891 | 13.464 |
| 2.50 | 3.6067 | 12.1825 | 25.790 |
| 3.00 | 4.3281 | 20.0855 | 49.402 |
| 3.50 | 5.0494 | 33.1155 | 94.632 |
| 4.00 | 5.7708 | 54.5982 | 181.272 |
| 5.00 | 7.2135 | 148.4132 | 665.14 |
A halving of perplexity corresponds to a fall of exactly ln 2 = 0.693 nats, or exactly 1 bit — the same step size everywhere on the scale.
Mistakes that make a perplexity number wrong or incomparable
- Confusing nats with bits. A loss of 2.5 is perplexity 12.18 in nats and 5.66 in bits. Both look reasonable, so nothing flags the error. Frameworks default to nats.
- Comparing across tokenisers. A larger vocabulary means fewer tokens for the same text and therefore a lower per-token loss, with no change in modelling quality. Convert to bits per character or per word before comparing.
- Comparing across corpora. Perplexity on curated web text and on transcribed speech are not the same measurement. State the evaluation set alongside the number.
- Ignoring how long documents are scored. Evaluating a long document in non-overlapping windows means every window's first tokens are predicted with almost no context, which inflates perplexity. A strided sliding window with a large stride overlap is the standard fix, and the stride must be reported.
- Averaging perplexities across documents. Perplexity is not linear, so the mean of per-document perplexities is not the corpus perplexity. Sum the log-probabilities over all tokens, divide by the token count, then exponentiate once.
- Reporting training rather than validation loss. Training loss on data the model has seen is not an estimate of anything generalisable, especially in the multi-epoch regime.
- Reading perplexity as answer quality. An instruction-tuned model often has higher perplexity on generic text than its base model and is much more useful. Different question, different metric.
Key terms
- Cross-entropy
- The mean of −log p(observed token | context) over the evaluation set. It measures how many nats or bits the model needs on average to encode the true next token.
- Nat
- The unit of information when logs are taken base e. One nat is 1 ÷ ln 2 = 1.4427 bits. Deep-learning frameworks use nats by default.
- Bits per character (BPC)
- Cross-entropy divided by the number of characters rather than tokens. Because it is tokeniser-independent, it is the standard way to compare models with different vocabularies.
- Uniform baseline
- The perplexity of a model that assigns equal probability to every vocabulary entry, which equals the vocabulary size. It is the reference point for 'has learned nothing'.
- Branching factor
- The intuitive reading of perplexity: the number of equally likely continuations the model's uncertainty is equivalent to at each step.
Where perplexity sits among evaluation metrics
Perplexity is the intrinsic metric — it scores the model's probability distribution directly, needs no labels beyond the text itself, and is cheap enough to run every few hundred steps. That makes it the right instrument for watching a pre-training run and for scaling-law work, where the relationship between loss, parameters and tokens is the object of study. If you are planning such a run, the LLM training compute calculator and the training time estimate calculator cover the compute side of the same trade.
Bits per character is the tokeniser-independent cousin and the metric to reach for when comparing across model families. Convert by multiplying bits per token by tokens per character, or equivalently dividing total bits by total characters. It is also the natural bridge to compression: a model with a given bits-per-character figure can drive an arithmetic coder at close to that rate, which is why language modelling and lossless compression are formally the same problem.
Extrinsic metrics answer a different question — is the model good at the thing you want? — and they diverge from perplexity often enough that neither substitutes for the other. Task benchmarks, human preference comparisons and pass rates on executable tests all measure behaviour after decoding, which perplexity never observes. The healthy pattern is perplexity for monitoring and for choosing among checkpoints of the same model, extrinsic evaluation for choosing among models to ship.
One historical note that explains the vocabulary: perplexity entered language modelling from speech recognition in the 1970s, where it was introduced as a measure of the difficulty of a recognition task — the average branching factor of the language being recognised. That framing survives, which is why the metric is still described as a branching factor rather than as an exponentiated loss.