What the test compares
Two groups, measured once each, with different people or items in each group: treatment and control, variant A and variant B, supplier one and supplier two. The two-sample t-test asks whether the gap between their averages is larger than the noise within them can comfortably explain.
The key word is independent. Every observation in group 1 must be unrelated to every observation in group 2. If the same subjects appear in both — before and after, left hand and right hand, matched pairs — this is the wrong test and it throws away the pairing that makes those designs powerful. Use the paired t-test instead.
The null hypothesis is that the two population means are equal. The test statistic is the observed difference divided by the standard error of that difference, and the standard error is built by adding variances: the uncertainty in x̄1 and the uncertainty in x̄2 combine as s₁²/n₁ + s₂²/n₂. Variances add for independent quantities, standard deviations do not, which is why the formula squares, adds, and takes the square root rather than adding the standard errors directly.
Notice the consequence: the group with the larger variance or the smaller sample dominates the standard error. Adding 200 more observations to an already-large group barely helps if the other group has 12. Balanced designs are efficient for exactly this reason.
Welch or pooled: which to use
The two versions differ in one assumption. The pooled test assumes both populations share a common variance, so it merges the two sample variances into one estimate sₚ², weighted by degrees of freedom, and uses n₁ + n₂ − 2 degrees of freedom. The Welch test makes no such assumption: it keeps the variances separate and pays for that with fractional degrees of freedom from the Welch–Satterthwaite approximation.
Welch is the better default, and this is not a close call in the modern literature. When the variances really are equal, Welch loses almost nothing — with equal sample sizes and equal variances it returns exactly the pooled answer, degrees of freedom included. When they are not equal, the pooled test's actual false-positive rate drifts away from the nominal alpha, and it drifts furthest when the group sizes are unequal and the larger variance sits in the smaller group.
The Welch degrees of freedom always lie between the smaller group's n − 1 and the pooled n₁ + n₂ − 2. They fall as the variances diverge: with two groups of 20, equal variances give 38 degrees of freedom, a 4:1 variance ratio gives 27.9, and a 25:1 ratio gives 20.5. That reduction is the correction — it widens the reference distribution to account for the fact that you cannot pool.
Do not choose between them by first running a variance test. A preliminary F test for equal variances is itself an inferential procedure, and conditioning the main test on its outcome distorts the error rate of both. Pick Welch in advance and stay with it. The calculator shows both because seeing them agree is reassuring, not because you should choose after the fact.
Worked example: 30 users against 32 users
Two onboarding flows are tested on separate cohorts. Flow A: 30 users, mean time-to-value 105 minutes, s = 12. Flow B: 32 users, mean 99 minutes, s = 14. These are the calculator's defaults. Run Welch, two-tailed, at α = 0.05.
- Variance of each mean. a = 12²/30 = 144/30 = 4.800; b = 14²/32 = 196/32 = 6.125.
- Standard error of the difference. √(4.800 + 6.125) = √10.925 = 3.3053 minutes.
- Difference. 105 − 99 = 6.000 minutes.
- t statistic. 6.000 / 3.3053 = 1.8153.
- Welch degrees of freedom. (10.925)² / [4.800²/29 + 6.125²/31] = 119.3556 / [0.794483 + 1.210181] = 119.3556 / 2.004664 = 59.54.
- p-value. The two-tailed area of t(59.54) beyond 1.8153 comes to about 0.075, which is above 0.05, so you fail to reject.
- Confidence interval. t(0.025, 59.54) ≈ 2.0003, so the interval for the difference is 6.000 ± 2.0003 × 3.3053 = 6.000 ± 6.612, or −0.61 to 12.61 minutes.
- Effect size. sₚ² = (29×144 + 31×196)/60 = 10,252/60 = 170.867, so sₚ = 13.072 and Cohen's d = 6.000/13.072 = 0.459.
The interval is the honest summary: flow B might be up to 12.6 minutes faster, or half a minute slower. A moderate effect size of d = 0.46 sitting on a non-significant p-value is the signature of an underpowered comparison, not of a null result. Detecting d = 0.46 at 80% power would need roughly 75 users per group.
Reading the output
Read the confidence interval for the difference first. It is in the units you measured — minutes, millimetres, dollars — and it tells you which effects your data rule out. A p-value above alpha with a narrow interval clustered around zero is a genuinely negative result. The same p-value with a wide interval means the study could not tell.
Then read Cohen's d, which strips out the sample size. Cohen's conventional labels put 0.2 as small, 0.5 as medium and 0.8 as large; they are rules of thumb from the behavioural sciences and should give way to whatever benchmark your field uses. Unlike t, d does not grow when you collect more data, which is exactly why it belongs in the report.
Check the sign against your group ordering. A negative t means group 1 has the smaller mean. The one-tailed alternatives in this calculator are stated as μ₁ > μ₂ and μ₁ < μ₂, so “right tail” asks whether group 1 is larger.
Finally, look at the two-method table. If the pooled and Welch rows lead to the same conclusion, the variance assumption is not driving your result and you can report either. If they disagree, report Welch and say so — the disagreement itself tells you the variances differ enough to matter.
How unequal variances cut the Welch degrees of freedom
| Variance ratio s₂² : s₁² | SD ratio | Welch df | Pooled df |
|---|---|---|---|
| 1 : 1 | 1.0 | 38.00 | 38 |
| 2 : 1 | 1.41 | 34.20 | 38 |
| 4 : 1 | 2.0 | 27.94 | 38 |
| 9 : 1 | 3.0 | 23.17 | 38 |
| 16 : 1 | 4.0 | 21.37 | 38 |
| 25 : 1 | 5.0 | 20.52 | 38 |
With equal group sizes the Welch df never fall below n − 1 = 19, which is the limit the table is approaching.
Pitfalls
- Using it on paired data. Before-and-after measurements on the same subjects belong in a paired t-test, which removes between-subject variation and is usually far more powerful.
- Choosing pooled or Welch after seeing which one is significant. Decide in advance. Conditioning the test on a preliminary variance test distorts the error rate.
- Running many two-group comparisons. With four groups there are six pairwise tests and the family-wise error rate climbs to roughly 1 − 0.95⁶ = 0.265, the exact figure being a little lower because the six comparisons share groups. Use a one-way ANOVA followed by a post-hoc procedure.
- Reporting only the p-value. The difference, its confidence interval and the effect size carry the information a reader needs; the p-value alone does not.
- Ignoring badly skewed data at small n. The t-test relies on approximately normal sampling distributions of the means. With small groups and heavy skew, use a rank-based test such as Mann-Whitney.
- Comparing two groups that were not independently assigned. The arithmetic works on any two columns of numbers; the causal reading requires randomisation.
Related tests and study planning
For one group against a fixed reference value, use the one-sample t-test calculator. For three or more groups, use the one-way ANOVA calculator — with exactly two groups, the ANOVA F equals this test's t squared when variances are pooled, and the p-values match to the last digit. For proportions rather than means, use the two-proportion z-test.
Before running an experiment, decide the smallest difference that would change your decision and check whether your planned sample can detect it. The statistical power calculator and the A/B test sample size calculator handle that, and the Cohen's d calculator converts between raw differences and standardised ones.
Welch's correction dates to a 1947 Biometrika paper, building on Satterthwaite's approximation for the degrees of freedom of a linear combination of variance estimates. Sixty years of simulation studies have since confirmed the practical recommendation: use Welch unless you have a specific reason not to.