What analysis of variance compares
Despite the name, ANOVA is a test about means. It compares three or more group averages by asking a question about variation: is the spread between the group means larger than you would expect given the spread within the groups?
Picture three teaching methods with test scores. If the three class averages differ by 12 points while scores inside each class scatter by 30, the class averages are unremarkable — you would expect that much separation from random assignment alone. If the averages differ by 12 points while scores inside each class scatter by 3, something is separating the groups. ANOVA formalises exactly that comparison, and the F statistic is the ratio of the two variances.
The null hypothesis is that every group was drawn from the same population, so all the population means are equal. The alternative is simply that they are not all equal — it does not say which one differs, or in which direction. That is the central limitation of ANOVA: a significant F tells you the means are not all the same, and nothing more. Identifying which pairs differ requires a post-hoc procedure such as Tukey's HSD, run after the F test rejects.
The obvious alternative — running a t-test on every pair — fails because of multiplicity. With four groups there are six pairwise comparisons, and if each runs at α = 0.05 the chance of at least one false positive rises to roughly 1 − 0.95⁶ = 0.265 when all the nulls are true. That figure treats the six comparisons as independent, which they are not — they share groups — but the direction and the rough size of the inflation are real. ANOVA answers the overall question with a single test at a single alpha.
How the sums of squares partition
Take every observation's deviation from the grand mean and square it; add them all up and you have the total sum of squares, SST. The insight behind ANOVA is that SST splits exactly into two pieces with no remainder.
SSB, the between-groups sum of squares, is Σnᵢ(x̄ᵢ − x̄)² — each group's squared distance from the grand mean, weighted by how many observations that group contains. It captures variation explained by which group an observation belongs to.
SSW, the within-groups sum of squares, is ΣΣ(x − x̄ᵢ)² — each observation's squared distance from its own group mean. It captures everything group membership does not explain, and it is the pooled error term.
The degrees of freedom split the same way: k − 1 for between, N − k for within, adding to N − 1 for the total. Dividing each sum of squares by its degrees of freedom gives a mean square, which is a variance estimate. Under H₀ both MSB and MSW estimate the same population variance, so their ratio should sit near 1. When the group means genuinely differ, MSB picks up that separation while MSW does not, so F climbs.
F cannot be negative, and its distribution is right-skewed with two separate degrees-of-freedom parameters. Only the upper tail is used: the p-value is the area beyond your F. You can look up the same tail area for any F with the p-value calculator.
Worked example: three teaching methods, five students each
Fifteen students are randomly assigned to three methods and sit the same test. Method A scores 82, 85, 88, 91, 79; method B scores 90, 94, 89, 96, 91; method C scores 78, 82, 80, 85, 75. These are the calculator's default values.
- Group means. A: 425/5 = 85. B: 460/5 = 92. C: 400/5 = 80.
- Grand mean. 1,285 / 15 = 85.6667.
- SSB. 5[(85−85.6667)² + (92−85.6667)² + (80−85.6667)²] = 5[0.4444 + 40.1111 + 32.1111] = 5 × 72.6667 = 363.3333.
- SSW. Group A deviations from 85 are −3, 0, 3, 6, −6, squaring and summing to 90. Group B gives 34, group C gives 58. SSW = 90 + 34 + 58 = 182.
- Degrees of freedom. Between: 3 − 1 = 2. Within: 15 − 3 = 12.
- Mean squares. MSB = 363.3333/2 = 181.6667. MSW = 182/12 = 15.1667.
- F. 181.6667 / 15.1667 = 11.978.
- Compare. The critical F(2, 12) at α = 0.05 is 3.885. Since 11.978 > 3.885, reject H₀.
- Effect size. SST = 363.3333 + 182 = 545.3333, so η² = 363.3333/545.3333 = 0.666 — two thirds of the variation in scores is associated with which method a student received.
The F test has established that the three methods do not all produce the same mean. It has not established that B beats A, or that A beats C. Those are three separate pairwise questions, and answering them without inflating the error rate needs Tukey's HSD or an equivalent adjusted comparison.
Reading the ANOVA table
Check F against its critical value, or equivalently the p-value against alpha — the two comparisons always agree, because the critical value is defined as the F whose upper tail area equals alpha. An F near 1 means the between-group spread is about what within-group noise alone would produce.
Then read eta-squared, because F on its own confounds effect size with sample size. η² is the share of total sum of squares attributable to group membership, so it runs from 0 to 1 and answers “how much” rather than “whether”. It is a descriptive statistic for the sample and is biased upward as an estimate of the population value, particularly with small groups; omega-squared corrects that bias and is preferred in published work.
Look at the group summaries before drawing any conclusion. A significant F driven by one extreme group with three observations tells a different story from one driven by an even ordering across five balanced groups. The group standard deviations also let you check the equal-variance assumption directly: this test pools all the within-group variation into one error term, which is only fair if the groups really do share a common variance.
Finally, remember what the design permits. ANOVA on randomly assigned groups supports a causal reading; ANOVA on groups that already existed — departments, countries, self-selected users — supports only an associational one, exactly as with correlation.
Critical values of F at α = 0.05
| df within | df between = 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| 5 | 6.61 | 5.79 | 5.41 | 5.19 | 5.05 |
| 10 | 4.96 | 4.10 | 3.71 | 3.48 | 3.33 |
| 12 | 4.75 | 3.89 | 3.49 | 3.26 | 3.11 |
| 20 | 4.35 | 3.49 | 3.10 | 2.87 | 2.71 |
| 30 | 4.17 | 3.32 | 2.92 | 2.69 | 2.53 |
| 60 | 4.00 | 3.15 | 2.76 | 2.53 | 2.37 |
Standard published F table. With one between-groups degree of freedom, F equals the square of the corresponding two-tailed t: 4.75 at df within = 12 is 2.179².
Assumptions and common errors
- Independence. Every observation must be independent of every other, both within and across groups. Repeated measures on the same subject need repeated-measures ANOVA, not this test.
- Equal variances. The pooled error term assumes the groups share a variance. As a working rule, a largest-to-smallest variance ratio beyond about 4 is a problem, especially with unequal group sizes; Welch's ANOVA relaxes the assumption.
- Approximate normality within groups. ANOVA is fairly robust to mild departures, but heavy skew or outliers at small group sizes distort both F and the p-value.
- Reading a significant F as identifying the biggest group. It does not. Follow with a post-hoc test that controls the family-wise error rate.
- Running ANOVA on two groups and reporting it as something different from a t-test. With k = 2, F equals t² exactly and the p-values are identical.
- Dropping outliers to reach significance. Removing points changes both the mean and the error term, and doing it after seeing the result invalidates the p-value entirely.
Where one-way ANOVA sits among the alternatives
With exactly two groups, use the two-sample t-test calculator: it gives the same answer, plus a signed difference and a confidence interval that ANOVA does not report. With one group against a fixed reference value, use the one-sample t-test calculator.
When the equal-variance assumption fails badly, Welch's ANOVA adjusts the degrees of freedom in the same way the Welch t-test does. When the data are ordinal or heavily skewed, the Kruskal-Wallis test ranks the observations and tests the ranks instead. When you have two grouping factors at once — method and school, dose and sex — you need two-way ANOVA, which also estimates the interaction between them.
For the effect size on its own, or to convert between effect-size measures when planning a study, see the Cohen's d calculator and the statistical power calculator. Analysis of variance was developed by R. A. Fisher at Rothamsted in the 1920s for agricultural field trials, which is why the vocabulary of treatments, blocks and plots still runs through the subject.