What F1 measures and why accuracy is not enough
F1 is the harmonic mean of precision and recall. Precision asks: of the cases the model flagged, how many were really positive? Recall asks: of the cases that were really positive, how many did the model flag? A model can max out either one on its own — predict positive exactly once and be right to get precision 1.0, predict positive on everything to get recall 1.0 — so neither is a score by itself. F1 combines them in a way that only rewards doing both.
The reason to use a harmonic mean rather than a plain average is that the harmonic mean is pulled toward the smaller number. Precision 1.0 with recall 0.02 averages to 0.51 but gives an F1 of 0.039. That is the behaviour you want from a single summary figure: it should not let a model hide a catastrophic weakness behind a strength.
Accuracy fails on imbalanced data for a related reason. If 1% of your cases are positive, a model that predicts “negative” unconditionally scores 99% accuracy, and it is worthless. F1 ignores the true-negative cell entirely, so that model scores zero. This is the single most common reason a review asks for F1 rather than accuracy, and it is why fraud, defect, churn and disease-screening problems are always reported with precision and recall side by side.
The formulas, and what each one is blind to
Everything comes out of four counts. Precision is TP ÷ (TP + FP): the denominator is everything the model called positive. Recall is TP ÷ (TP + FN): the denominator is everything that actually was positive. Specificity is TN ÷ (TN + FP), which is recall for the negative class. Accuracy is (TP + TN) ÷ N.
F1 written out in cells is 2·TP ÷ (2·TP + FP + FN), which makes its one blind spot obvious: TN never appears. Add a million true negatives to your test set and F1 does not move. That is a feature when the negative class is uninteresting background, and a bug when getting the negatives right is part of the job.
F-beta generalises F1 by weighting: Fβ = (1 + β²)·P·R ÷ (β²·P + R). The parameter is defined so that β is the number of times you consider recall as important as precision — so β = 2 weights recall four times as heavily in the algebra, because the weight enters as β². Use β > 1 when a miss is expensive (a missed tumour, a missed intrusion) and β < 1 when a false alarm is expensive (a wrongly blocked payment, an auto-flagged account).
Balanced accuracy is (recall + specificity) ÷ 2, the average of the two class-wise recalls. It corrects accuracy for imbalance but says nothing about precision, so a model with a huge false-positive count can still score well when negatives are plentiful. The Matthews correlation coefficient is the only one of these that uses all four cells symmetrically; it is the Pearson correlation between the predicted and true label vectors, which is why it runs from −1 to +1 with 0 at chance rather than from 0 to 1.
Worked example: 1,000 transactions, 120 of them fraudulent
A fraud model is evaluated on 1,000 transactions. It flags 100, of which 90 are genuinely fraudulent. There were 120 frauds in total. That fixes all four cells: TP = 90, FP = 10, FN = 120 − 90 = 30, TN = 1,000 − 90 − 10 − 30 = 870.
- Precision. 90 ÷ (90 + 10) = 90 ÷ 100 = 0.9000. Nine out of ten flags were real.
- Recall. 90 ÷ (90 + 30) = 90 ÷ 120 = 0.7500. Three quarters of the fraud was caught.
- F1. 2 × 0.9 × 0.75 = 1.35 on top; 0.9 + 0.75 = 1.65 underneath; 1.35 ÷ 1.65 = 0.8182. Note it sits below the arithmetic mean of 0.825, pulled toward the weaker recall.
- Accuracy. (90 + 870) ÷ 1,000 = 0.9600 — much friendlier than F1, because the 870 true negatives dominate a count that F1 never sees.
- Specificity. 870 ÷ (870 + 10) = 870 ÷ 880 = 0.9886.
- Balanced accuracy. (0.75 + 0.9886) ÷ 2 = 0.8693.
- MCC. The numerator is 90×870 − 10×30 = 78,300 − 300 = 78,000. The denominator is √(100 × 120 × 880 × 900) = √9,504,000,000 = 97,488.46. So MCC = 78,000 ÷ 97,488.46 = 0.8001.
Now suppose the business decides a missed fraud costs about four times a false alarm. That is β = 2: F₂ = 5 × 0.9 × 0.75 ÷ (4 × 0.9 + 0.75) = 3.375 ÷ 4.35 = 0.7759. The score falls relative to F1 because the model’s weaker side is recall, and β = 2 puts more weight there.
How to read the numbers you get back
There is no universal threshold for a “good” F1, because the value depends on the base rate and the difficulty of the task as much as on the model. An F1 of 0.55 can be excellent on a 0.5%-prevalence fraud problem and terrible on balanced sentiment classification. The only interpretations that transfer are relative ones: against a baseline, against the previous model, and against the cost structure of the decision.
Build the baseline explicitly. For a set with prevalence p, a classifier that flags everything gets precision p, recall 1, and F1 = 2p ÷ (1 + p). At 10% prevalence that is 0.182 — so an F1 of 0.25 on such a set is barely better than predicting positive at random with no model at all. Quote your F1 next to that number and the reader can judge it.
Read precision and recall separately before you read F1, because two very different models can share an F1. Precision 0.9 with recall 0.75 and precision 0.75 with recall 0.9 both give 0.8182, and they behave nothing alike in production: one under-flags, the other over-flags. F1 is a ranking tool for model selection, not a description of behaviour.
Use MCC when you want one number that cannot be gamed by class imbalance. Because it uses all four cells, a model that gets a good F1 by carpet-bombing the positive class shows up immediately: in the rare-positive row of the table below, F1 is 0.18 and MCC is 0.20 while accuracy reads a comfortable 0.91. A negative MCC is a specific, actionable signal — the predictions are anti-correlated with the labels, and the usual cause is an inverted label mapping rather than a bad model.
The same metrics on five different confusion matrices
| TP / FP / FN / TN | Precision | Recall | F1 | Accuracy | Balanced acc. | MCC |
|---|---|---|---|---|---|---|
| 90 / 10 / 30 / 870 | 0.9000 | 0.7500 | 0.8182 | 0.9600 | 0.8693 | 0.8001 |
| 18 / 2 / 2 / 78 | 0.9000 | 0.9000 | 0.9000 | 0.9600 | 0.9375 | 0.8750 |
| 50 / 0 / 50 / 900 | 1.0000 | 0.5000 | 0.6667 | 0.9500 | 0.7500 | 0.6882 |
| 1 / 1 / 8 / 90 | 0.5000 | 0.1111 | 0.1818 | 0.9100 | 0.5500 | 0.2047 |
| 0 / 0 / 20 / 80 | undefined | 0.0000 | undefined | 0.8000 | 0.5000 | undefined |
Rows one and two share an accuracy of 0.9600 and differ by 0.08 in F1. The last row is the degenerate case every metric except accuracy and specificity refuses to score: with no predicted positives, three of the four MCC marginals collapse and precision is 0 ÷ 0.
Macro, micro and weighted F1 for more than two classes
With more than two classes you compute the metrics one class at a time — treating that class as positive and everything else as negative — and then aggregate. The aggregation choice changes the answer substantially, so it belongs in the report alongside the number.
Macro F1 is the unweighted mean of the per-class F1 scores. Every class counts equally regardless of size, so a small class you handle badly drags the score down hard. Micro F1 pools the cells first — sum TP, FP and FN across classes, then compute one F1 — so large classes dominate. Weighted F1 averages the per-class scores using class support as weights, landing between the two.
Work a three-class example. Class A has TP 50, FP 10, FN 5; class B has TP 30, FP 20, FN 10; class C has TP 5, FP 5, FN 20. Using F1 = 2·TP ÷ (2·TP + FP + FN): class A is 100 ÷ 115 = 0.8696, class B is 60 ÷ 90 = 0.6667, class C is 10 ÷ 35 = 0.2857. Macro F1 is (0.8696 + 0.6667 + 0.2857) ÷ 3 = 0.6073.
Micro pools instead: TP = 85, FP = 35, FN = 35, so micro precision = micro recall = 85 ÷ 120 = 0.7083 and micro F1 = 0.7083. In single-label multiclass every mistake is simultaneously one class’s false positive and another class’s false negative, so the FP and FN totals are always equal and micro F1 equals plain accuracy — here 85 correct out of 120 predictions. The 0.10 gap between macro and micro is entirely the story of class C: 25 cases, handled at 0.29, given a third of the weight under macro and a fifth under micro.
Mistakes that make a reported F1 meaningless
- Quoting F1 without the base rate. The same number means different things at 1% and 50% prevalence. Report prevalence and the all-positive baseline of 2p/(1+p) alongside it.
- Tuning the threshold on the test set. F1 depends on the decision threshold, so picking the threshold that maximises F1 on the same data you report it from is leakage. Tune on validation, report on test.
- Averaging F1 across folds. The mean of per-fold F1 scores is not the F1 of the pooled predictions, because F1 is not linear in the cells. Pool the confusion matrices, then compute once.
- Silently treating undefined as zero. When a model predicts no positives, precision is 0 ÷ 0. Most libraries substitute 0 and emit a warning; if that warning is suppressed, a broken run looks like a merely bad one.
- Comparing macro F1 from one paper with micro F1 from another. They answer different questions and can differ by ten points on the same predictions, as the three-class example above shows.
- Using F1 when true negatives matter. F1 cannot see the TN cell at all. If correctly clearing negatives is part of the product’s value, quote balanced accuracy or MCC instead.
- Reporting a single point when you have scores. If the model outputs probabilities, threshold-free summaries such as average precision or the area under the precision-recall curve carry far more information than F1 at one arbitrary cut.
Where F1 sits among the alternatives
Medicine got here first and uses different names for the same arithmetic. Recall is sensitivity and specificity is the negative-class recall, both handled by the sensitivity and specificity calculator; precision is the positive predictive value computed by the PPV and NPV calculator. The clinical literature prefers likelihood ratios precisely because, unlike precision and F1, they do not change when the prevalence of the condition changes — a property worth having when the same model is deployed on populations with different base rates.
If you are comparing two models rather than describing one, remember that a difference in F1 on a finite test set is an estimate with uncertainty attached, and the same significance machinery used for an A/B test applies to the comparison. And if the classifier in question is a neural network you are about to deploy, the metrics here answer whether it is good enough while the GPU memory calculator and the parameter count calculator answer whether you can afford to serve it.
The F-measure itself is old: van Rijsbergen introduced the effectiveness measure that F is derived from in Information Retrieval in 1979, in the context of ranking documents, where the true-negative cell is meaningless because it counts every document in the collection that the query correctly ignored. That origin explains the metric’s one blind spot better than any later justification does.