What statistical significance does and does not tell you
Significance answers one narrow question: if the two pages converted at exactly the same rate, how often would random assignment alone produce a gap at least this large? That probability is the p-value. A p-value of 0.03 means a gap this big would turn up 3% of the time by pure luck.
Notice what is missing from that sentence. The p-value is not the probability that the variant is better. It is not the probability that you are right. It says nothing about how big the improvement is, and nothing about whether the improvement is worth the engineering cost. A p-value of 0.001 on a 0.2% lift is a rock-solid measurement of something too small to matter.
This is why the confidence interval on the difference is the more useful output. A test that returns "p = 0.04, lift 25%, interval 1.3 to 8.7 percentage points" is telling you the variant is probably better and probably better by somewhere between a little and a lot. A test that returns "p = 0.31, lift 12%, interval −4 to +9 percentage points" is telling you that you have learned almost nothing: a meaningful loss is still on the table.
One more distinction worth burning in. A non-significant result is inconclusive, not a tie. Failing to reject the null hypothesis is not the same as showing the two pages are equivalent. To claim equivalence you need the whole interval to sit inside a band you consider negligible, which usually takes more traffic than proving a difference.
How the two-proportion z-test works
The test measures a distance and divides it by the noise in that distance. The distance is the observed gap between rates, p2 − p1. The noise is the standard error of that gap. The ratio, z, is the number of standard errors the gap covers, and a normal table turns that into a probability.
The pooled rate is the interesting part. The null hypothesis states the two rates are equal, so under the null there is only one true rate — and the best estimate of it uses every observation from both arms: p̄ = (x1 + x2) ÷ (n1 + n2). That is why the denominator uses p̄ rather than each arm's own rate. It is a deliberate choice: you compute the noise as if the null were true, then ask how unlikely your observation is.
The variance of a proportion is p(1−p). There is no separate spread parameter to estimate the way there is with a t-test — the rate determines its own variance. This is also why extreme rates are quieter than middling ones: at 50% the variance is 0.25, at 3% it is only 0.029.
Dividing by 1/n₁ + 1/n₂ makes the smaller arm dominate. If one arm has 100,000 visitors and the other 1,000, the standard error is set almost entirely by the small arm. Piling extra traffic into the control does very little; you have to grow both.
For the confidence interval the calculator switches to the unpooled standard error, √[p1q1/n1 + p2q2/n2]. Once you are estimating an effect rather than testing a null, you no longer assume the rates are equal, so each arm contributes its own variance. The two standard errors are close in practice, which is why a p-value just under 0.05 sometimes pairs with an interval whose bound sits a hair the wrong side of zero.
Worked example: 200 conversions from 1,000 versus 250 from 1,000
Your control page turned 1,000 visitors into 200 signups. The variant turned 1,000 visitors into 250. Work it through by hand at a 0.05 threshold, two-sided.
- Rates. p1 = 200 ÷ 1,000 = 0.2000. p2 = 250 ÷ 1,000 = 0.2500. The gap is 0.0500, or 5 percentage points.
- Pooled rate. p̄ = (200 + 250) ÷ (1,000 + 1,000) = 450 ÷ 2,000 = 0.2250.
- Pooled variance factor. p̄(1 − p̄) = 0.2250 × 0.7750 = 0.174375.
- Sample-size factor. 1/1,000 + 1/1,000 = 0.002.
- Standard error. √(0.174375 × 0.002) = √0.00034875 = 0.0186748.
- z-score. 0.0500 ÷ 0.0186748 = 2.6774.
- p-value. The upper tail beyond z = 2.6774 is 0.0037099. Two-sided, double it: p = 0.00742. Confidence = 1 − 0.00742 = 99.26%.
- Relative lift. 0.0500 ÷ 0.2000 = 25.0%.
- Interval. Unpooled SE = √(0.20×0.80/1,000 + 0.25×0.75/1,000) = √(0.00016 + 0.0001875) = √0.0003475 = 0.0186414. Half-width = 1.959964 × 0.0186414 = 0.036536. So the 95% interval on the difference is 0.0500 ± 0.0365 = 1.35 to 8.65 percentage points.
Read that last line carefully, because it is the honest summary of the experiment. In relative terms the interval runs from a 6.7% lift to a 43.3% lift. You are confident the variant wins; you have almost no idea by how much. Anyone who reports "we found a 25% lift" and plans revenue on it is over-reading a 2,000-visitor test.
How to read the result and decide what to ship
Work down these four checks in order. Skipping straight to the p-value is how bad decisions get made.
1. Is the data trustworthy? Check the split first. If you intended 50/50 and got 52/48 on a large test, that is sample ratio mismatch and it means visitors were assigned or logged incorrectly. Fix the instrumentation and rerun; a p-value computed on broken assignment is meaningless. This calculator flags a split that departs significantly from even.
2. Are the counts large enough for the math? The z-test approximates a binomial with a normal curve, and that approximation needs a reasonable number of successes and failures in every cell. Below about 30 conversions in an arm, treat the p-value as indicative and prefer an exact test. Below five expected outcomes in any cell, the z-test simply does not apply.
3. Does the interval clear your indifference threshold? Decide before the test how much lift would justify shipping — engineering time, maintenance, risk. If your threshold is 3 percentage points and the interval runs from 1.3 to 8.7, the test has not established that the change clears the bar even though it is significant. Significance is about zero; your decision is about your threshold.
4. Was the sample fixed in advance? If you have been watching daily and are reading this because the number finally went green, your real false-positive rate is far above the nominal 5%. Plan the sample with the A/B test sample size calculator and analyse once.
When the answer is significant and the interval clears your threshold, convert it to money before you celebrate: multiply the lower bound of the interval by your traffic and value per conversion. The revenue per visitor calculator does that arithmetic, and planning on the lower bound rather than the point estimate is what keeps forecasts honest.
z-score, p-value and confidence level reference
| z-score | Two-sided p | One-sided p | Confidence (two-sided) |
|---|---|---|---|
| 1.282 | 0.200 | 0.100 | 80.0% |
| 1.645 | 0.100 | 0.050 | 90.0% |
| 1.960 | 0.050 | 0.025 | 95.0% |
| 2.170 | 0.030 | 0.015 | 97.0% |
| 2.326 | 0.020 | 0.010 | 98.0% |
| 2.576 | 0.010 | 0.005 | 99.0% |
| 3.090 | 0.002 | 0.001 | 99.8% |
| 3.291 | 0.001 | 0.0005 | 99.9% |
A z of 1.96 is the familiar 95% bar. Note how quickly the tail thins: adding 0.6 to the z-score takes you from p = 0.05 to p = 0.01.
A significant p-value from a peeked-at test is not evidence
The 5% false-positive rate of this test holds for exactly one analysis at a pre-committed sample size. If you evaluate every morning and stop the first time the bar goes green, you have run many tests and reported the luckiest one. Repeated-looks simulations show the real false-positive rate climbing to several times the nominal level.
Two legitimate fixes exist. Fix the sample size in advance and analyse once. Or adopt a method designed for continuous monitoring — a group-sequential design with alpha-spending boundaries, or an always-valid sequential test. What does not work is applying a fixed-horizon p-value to a test you have been watching.
Ways this calculation gets misread
- Treating the p-value as the probability the variant wins. It is the probability of your data under the assumption that the variant does not win. Those are different quantities, and only a Bayesian analysis gives you the first one.
- Calling a non-significant test a tie. An inconclusive result usually has an interval wide enough to contain a real loss. If you need to prove two pages are equivalent, set an equivalence margin and check the whole interval fits inside it.
- Reporting the point estimate as the expected lift. The observed 25% in the example above could easily be a true 8%. Forecast from the lower bound of the interval, not the middle.
- Mixing units between arms. Sessions in one arm and users in the other, or different date ranges, breaks the comparison entirely. Both counts must come from the same definition and the same window.
- Running the test on visitors who never saw the change. Diluting both arms with unexposed traffic shrinks the measurable effect and can turn a real win into a null result.
- Checking twenty metrics and reporting the one that moved. At α = 0.05, twenty independent metrics give you a 64% chance of at least one false positive. Nominate the primary metric before launch.
- Ignoring novelty effects. A visibly new design can lift or depress conversion for its first week purely because it is unfamiliar. Look at the daily trend, not just the total.
Assumptions and limits of this test
The z-test assumes each visitor is an independent observation with a fixed conversion probability, that a visitor belongs to exactly one arm for the whole test, that the two arms were assigned at random, and that the conversion rate is stable across the window. It also assumes the counts are large enough for the normal approximation to a binomial.
Independence is the assumption most often violated on real sites. A visitor who returns on a second device is counted twice, often in different arms. Sessions instead of users correlate within a person. Both cases understate the true variance, which makes the p-value look better than it should. If your tool reports both a user-level and a session-level number, the user-level one is the valid comparison.
The test is also silent about mechanism and durability. It cannot tell you whether a lift comes from a genuinely better page, from a novelty effect that fades, or from an outage in the control during the test window. Segmenting the daily series is the standard check: a real effect is roughly stable day to day, a novelty effect decays, and an instrumentation problem starts abruptly.
For continuous outcomes — revenue per visitor, order value, time on task — this calculation does not apply at all. Those need a two-sample t-test or a bootstrap, and they need far more traffic because a handful of large orders can dominate the variance. If you want the general statistical form of this test outside a marketing funnel, the two-proportion z-test calculator covers it, and the proportion confidence interval calculator handles a single rate.
Where this sits among the alternatives
The pooled two-proportion z-test is the default because it is simple, well understood, and matches what commercial platforms report. Several alternatives are better in specific situations.
Fisher's exact test computes the exact hypergeometric probability instead of approximating, and is the right choice when a cell count is small. It is conservative and slower but never wrong for small samples.
The chi-square test of independence on the 2×2 table gives a statistic equal to z², so for two groups it is the identical test in different clothing. It generalises to more than two variations, at the cost of telling you only that something differs.
Bayesian A/B analysis reports the probability that the variant is better and the expected loss from choosing wrongly. Many teams find those quantities easier to act on than a p-value, and they do not break under monitoring the way a fixed-horizon p-value does — though they still need enough data to be decisive.
Sequential tests keep the error rate valid while you watch. If your organisation cannot leave a dashboard alone, this is the honest answer rather than a discipline problem to be solved by exhortation.
Whatever method you use, the funnel context matters as much as the statistics. Before choosing what to test, find the step that is actually leaking with the sales funnel drop-off calculator, quantify checkout losses with the cart abandonment rate calculator, and measure your starting point cleanly with the conversion rate calculator.
Key terms
- p-value
- The probability of observing a difference at least as extreme as yours, assuming the two variations convert at the same rate. Small means surprising under the null, nothing more.
- Confidence level
- One minus the p-value, expressed as a percentage. "96% confidence" is simply p = 0.04 restated, and carries no extra information.
- Pooled proportion
- The combined conversion rate across both arms, used to build the standard error under the null hypothesis that the two rates are equal.
- Confidence interval on the difference
- The range of true differences consistent with your data at the chosen level. If it excludes zero, the test is significant; its width tells you how precisely you have measured the effect.
- Sample ratio mismatch
- A statistically significant departure of the observed traffic split from the intended split. Almost always a bug, and a reason to discard the result.
- Equivalence testing
- The procedure for showing two variations are effectively the same: define a margin of indifference and require the whole confidence interval to fall inside it.