What the confusion matrix contains and why accuracy is not enough
A confusion matrix is the complete record of a binary classifier's behaviour on a labelled test set: four counts, from which every scalar metric is derived. Nothing is lost in the matrix and a great deal is lost in any single number you compute from it, which is why the matrix is the thing to report and the scalars are summaries of it.
Accuracy — (TP + TN) ÷ n — is the summary everyone reaches for, and it fails in a specific, predictable way. When one class is rare, a model that always predicts the majority class scores the majority class's share. At 1% prevalence, a model that outputs "negative" for every input scores 99% accuracy, and it has learned nothing: it would score exactly the same on random data. That model's balanced accuracy is 50%, its MCC is 0 and its kappa is 0, which is the correct answer three times over.
The fix is not to abandon accuracy but to read it next to a metric that treats the two classes symmetrically. Balanced accuracy averages recall and specificity, so each class contributes equally regardless of how many cases it has. MCC is the correlation between predictions and labels, using all four cells. Cohen's kappa subtracts the agreement you would expect from chance given the marginal totals, and rescales what is left.
These three answer slightly different questions, and they disagree in informative ways. Reporting all of them alongside the raw matrix is more useful than arguing about which single number is best.
Reading MCC, balanced accuracy and kappa as formulas
MCC is the Pearson correlation coefficient applied to two binary variables: the predicted label and the true label. Its numerator, TP·TN − FP·FN, is a determinant — it is positive when the diagonal dominates and negative when the off-diagonal does. The denominator is the geometric mean of all four marginal totals, which normalises the result into [−1, +1]. Because every cell appears, MCC cannot be inflated by one large cell: a model must do well on both classes to score well.
The symmetry has a practical consequence worth knowing. If you swap which class you call positive — exchange TP with TN and FP with FN — MCC is unchanged, and so is accuracy and balanced accuracy. Precision, recall and F1 all change, sometimes drastically. That is a good reason to prefer MCC when the choice of positive class is arbitrary, and to prefer recall or precision when it is not.
Balanced accuracy is the simplest correction: recall measures performance on actual positives, specificity on actual negatives, and averaging them gives each class equal weight. It is exactly the accuracy you would measure on a hypothetical test set with equal class sizes. Its floor is 50% for any constant classifier, which makes 50% the number to compare against rather than 0%.
Cohen's kappa asks a different question: how much better than chance is this? Expected chance agreement pe is computed from the marginals — multiply the share predicted positive by the share actually positive, add the same product for the negatives — and kappa is the fraction of the remaining headroom that the model captured. It is 0 when the model agrees with the labels exactly as often as its own marginal rates predict, and 1 for perfect agreement. Kappa's denominator vanishes when pe = 1, which happens when everything is predicted and labelled the same class; that case reports as undefined here.
Worked example: 90 TP, 30 FP, 10 FN, 870 TN
A fraud model is evaluated on 1,000 transactions, of which 100 are truly fraudulent. It catches 90 of them, misses 10, and raises 30 false alarms.
- Totals. n = 90 + 30 + 10 + 870 = 1,000. Actual positives = 90 + 10 = 100. Actual negatives = 30 + 870 = 900. Predicted positives = 90 + 30 = 120. Predicted negatives = 10 + 870 = 880.
- Accuracy. (90 + 870) ÷ 1,000 = 960 ÷ 1,000 = 96.00%.
- Recall. 90 ÷ 100 = 90.00%. Specificity. 870 ÷ 900 = 96.67%. Precision. 90 ÷ 120 = 75.00%.
- Balanced accuracy. (0.9000 + 0.9667) ÷ 2 = 93.33%.
- MCC numerator. 90 × 870 − 30 × 10 = 78,300 − 300 = 78,000.
- MCC denominator. √(120 × 100 × 900 × 880) = √(12,000 × 792,000) = √9,504,000,000 = 97,488.46.
- MCC. 78,000 ÷ 97,488.46 = 0.8001.
- Chance agreement. pe = (120 × 100 + 880 × 900) ÷ 1,000² = (12,000 + 792,000) ÷ 1,000,000 = 0.8040.
- Kappa. (0.9600 − 0.8040) ÷ (1 − 0.8040) = 0.1560 ÷ 0.1960 = 0.7959.
The gap between 96.00% accuracy and 0.80 MCC is the useful signal. Accuracy is flattered by the 870 true negatives, which are easy; MCC and kappa both land near 0.80 because they discount agreement that the class balance would have produced anyway. Note that a model predicting "not fraud" every time would have scored 90.0% accuracy here — so 96.0% is only 6 points better than doing nothing, while 0.80 MCC is 0.80 better than doing nothing.
How to read each number and which one to optimise
Start by computing the majority-class baseline: the larger of (TP + FN) and (FP + TN), divided by n. That is the accuracy of a model with no skill, and any accuracy figure must be read against it rather than against 100%. The calculator raises a warning when that baseline is 90% or higher, because that is where accuracy stops carrying information.
MCC has no universally agreed thresholds, and any table claiming otherwise is a convention rather than a fact. What is defined is the scale: 0 is the expected value for random or constant prediction, 1 is perfect, and negative values mean the predictions are anti-correlated with the labels — a model scoring −0.6 would be substantially useful if you inverted its output. Kappa is often read against Landis and Koch's descriptive bands, which label 0.41–0.60 as moderate, 0.61–0.80 as substantial and above 0.81 as almost perfect; those labels come from inter-rater agreement in medical research and were proposed as arbitrary but useful, not derived.
Which metric to optimise depends on the asymmetry of your errors, and that is a product decision rather than a statistical one. When a miss is far more costly than a false alarm — disease screening, fraud detection, safety filtering — recall dominates and you accept the precision cost. When a false alarm is expensive — automated enforcement, irreversible actions — precision dominates. When the two are comparable and classes are imbalanced, MCC is the best single summary. The F1 score calculator covers the precision/recall trade explicitly, including weighted Fβ variants.
One caution about all of these: they describe a classifier at a single decision threshold. Almost every model outputs a score, and the confusion matrix is what falls out after you pick a cut-off. Changing the threshold moves every number on this page. If you are comparing models rather than deployments, threshold-free summaries such as the area under the ROC or precision-recall curve are the fairer comparison.
Five matrices that show where accuracy and MCC diverge
| TP | FP | FN | TN | Accuracy | Balanced acc. | MCC | Kappa |
|---|---|---|---|---|---|---|---|
| 450 | 50 | 50 | 450 | 90.0% | 90.0% | 0.800 | 0.800 |
| 90 | 30 | 10 | 870 | 96.0% | 93.3% | 0.800 | 0.796 |
| 0 | 0 | 10 | 990 | 99.0% | 50.0% | 0.000 | 0.000 |
| 100 | 400 | 0 | 500 | 60.0% | 77.8% | 0.333 | 0.200 |
| 100 | 0 | 0 | 900 | 100.0% | 100.0% | 1.000 | 1.000 |
Row 3 is the classifier that predicts one class for everything: the highest accuracy in the table bar the perfect model, and the worst MCC. Row 4 is the opposite trap — the lowest accuracy in the table, but a model that catches every positive.
Mistakes that make these metrics misleading
- Reporting accuracy without the class balance. An accuracy figure is uninterpretable until the reader knows the majority-class baseline it is being compared against.
- Transposing the matrix. Swapping FP and FN leaves accuracy and MCC unchanged but inverts precision and recall, so a transposed matrix produces plausible-looking numbers that describe a different model.
- Comparing metrics across different thresholds. Every number here is threshold-specific. Two models compared at different operating points are not being compared at all.
- Tuning the threshold on the test set. Choosing the cut-off that maximises MCC on the same data you report it from turns an estimate into an upper bound. Use a validation split.
- Treating kappa bands as objective. The 'substantial' and 'almost perfect' labels were proposed as convenient descriptors for inter-rater agreement, not derived from a distribution, and kappa itself is sensitive to prevalence.
- Ignoring the interval on small samples. On 50 cases, one reclassification moves accuracy by two points. Report a confidence interval, or a larger test set.
- Reading MCC = 0 as a measurement when a margin is zero. If a whole row or column is empty, MCC's denominator vanishes and the 0 is a convention, not a score.
Key terms
- Prevalence
- The share of cases that are actually positive, (TP + FN) ÷ n. It is a property of your test set, not of the model, and it drives how misleading accuracy is.
- Sensitivity / recall / TPR
- Three names for TP ÷ (TP + FN): the share of actual positives the model catches. Medicine says sensitivity, information retrieval says recall, signal detection says true positive rate.
- Specificity / TNR
- TN ÷ (TN + FP): the share of actual negatives the model correctly clears. Its complement, 1 − specificity, is the false positive rate plotted on a ROC curve.
- Youden's J
- recall + specificity − 1, also called informedness. It is balanced accuracy rescaled to put chance at 0 instead of 0.5, and it is a common criterion for choosing a threshold.
- Chance agreement (pₑ)
- The agreement rate two independent raters would reach given the observed marginal rates. Cohen's kappa is the share of the remaining headroom above pₑ that the model actually achieved.
Beyond the 2×2 matrix
Multi-class problems produce a k×k matrix, and the standard approach is to reduce it to a set of one-versus-rest 2×2 matrices and then average. How you average matters: macro averaging treats every class equally regardless of size, micro averaging pools all the cells first and therefore weights by frequency (and for single-label problems collapses to accuracy), and weighted averaging sits between them. MCC generalises directly to the multi-class case through the Gorodkin formulation, which is what most libraries implement.
For probabilistic outputs, the confusion matrix throws away information you may want to keep. Calibration — whether a predicted probability of 0.7 corresponds to a 70% hit rate — is invisible here, and a model can have excellent MCC and badly miscalibrated probabilities. Brier score and reliability diagrams cover that gap. For ranking quality independent of any threshold, use the area under the ROC curve, or the area under the precision-recall curve when positives are rare, since the latter is far more sensitive at low prevalence.
In clinical and diagnostic settings, the same four cells feed a different vocabulary and a different set of derived quantities: positive and negative predictive values are computed by the PPV and NPV calculator, the underlying pair by the sensitivity and specificity calculator, and the prevalence-independent versions of the same information by the diagnostic likelihood ratio calculator. For the association between exposure and outcome rather than the performance of a classifier, the odds ratio calculator works from the same table.
For language models, classification metrics are only part of evaluation; likelihood-based measures such as the one in the perplexity calculator score the model's probability distribution rather than a thresholded decision.