What a sample size actually buys you
A sample size is a purchase of statistical resolution. You are buying the ability to see an effect of a chosen size, with a chosen chance of seeing it if it is real, and a chosen chance of hallucinating it if it is not. Four numbers are welded together: the baseline rate p₁, the smallest effect you want to detect, the significance level α, and the power 1−β. Pin down any three and the fourth is arithmetic.
The reason to pin them down before launch is that the alternative corrupts the test. If you check the result every morning and stop the moment p drops below 0.05, you are not running one test at 5% risk — you are running one test per look, and the chance that at least one of them crosses the line by luck climbs far above 5%. Sequential-testing research on this “peeking” problem shows the effective false-positive rate several times the nominal level for realistic monitoring schedules. A fixed-horizon test with a pre-computed sample size is the cheapest fix: decide the stopping point once, honour it, and your 95% actually means 95%.
The number that does the most work here is the minimum detectable effect (MDE), and it is a business decision, not a statistical one. Ask what lift would be big enough to justify shipping the change, retraining the team, and carrying the code. If a 3% relative lift on checkout would pay for itself in a month, your MDE is 3% and you need a lot of traffic. If nothing under 15% would change your roadmap, say 15% and finish in a fortnight.
The formula, term by term
The numerator carries two z values because you are controlling two different errors on two different distributions. Under the null hypothesis both arms convert at the same pooled rate p̄, so the sampling variance of the difference is 2·p̄(1−p̄); the term z(1−α/2)·√ that is how far apart two identical arms can drift by chance before you would call it significant. Under the alternative hypothesis the arms convert at p₁ and p₂, so the variance is p₁(1−p₁) + p₂(1−p₂); the term z(1−β)·√ that is how much extra room you need so that a real difference lands past the threshold most of the time. Add the two distances, divide by the effect you must span, square it.
Three consequences follow directly from the algebra, and they explain almost every surprised reaction to a sample-size estimate.
The denominator is squared, so resolution is expensive. Halving your MDE multiplies the sample by four. Detecting a 5% relative lift instead of a 20% one on a 5% baseline moves you from 8,158 visitors per arm to 122,124 — fifteen times the traffic for a target four times smaller.
Rare events cost more. For a fixed relative lift, the sample scales roughly with (1−p₁)/p₁. Going from a 10% baseline to a 1% baseline multiplies the requirement by about eleven. This is why testing on a step deep in the funnel is so slow, and why moving your primary metric up to a higher-frequency step is often the difference between a two-week test and a two-quarter one.
Power is not symmetric with significance. Raising confidence from 95% to 99% on our example takes the sample from 8,158 to 12,139 per arm, because z(0.995) = 2.575829 replaces 1.959964; raising power from 80% to 90% takes it to 10,921. Both cost traffic, but the default 80% power means you accept a one-in-five chance of missing a real effect of exactly your MDE size. If missing a winner is expensive, buy power before you buy confidence.
Worked example: 5% baseline, 20% relative lift, 95% confidence, 80% power
Your checkout converts at 5.0%. You will ship a redesign only if it lifts conversion by at least 20% relative — to 6.0%. You run a two-arm test at α = 0.05 two-sided and 80% power, with 2,000 visitors a day entering the experiment.
- Set the two rates. p₁ = 0.05, p₂ = 0.05 × 1.20 = 0.06. The absolute effect δ = 0.01.
- Look up the two z values. z(1−0.05/2) = z(0.975) = 1.959964. z(0.80) = 0.841621.
- Null variance. p̄ = (0.05 + 0.06)/2 = 0.055. 2 × 0.055 × 0.945 = 0.10395. √0.10395 = 0.322413.
- First term. 1.959964 × 0.322413 = 0.631914.
- Alternative variance. 0.05 × 0.95 = 0.0475; 0.06 × 0.94 = 0.0564; sum = 0.1039. √0.1039 = 0.322335.
- Second term. 0.841621 × 0.322335 = 0.271283.
- Add and divide by δ. (0.631914 + 0.271283) ÷ 0.01 = 90.3197.
- Square it. 90.3197² = 8,157.7 → 8,158 visitors per variation.
- Total and duration. 8,158 × 2 = 16,316 visitors. 16,316 ÷ 2,000 = 8.16 → 9 days, which you round up to 14 so the test covers two complete weekly cycles.
Sanity-check the conversion counts before you commit: 8,158 × 5% ≈ 408 conversions in control and about 490 in the variant if the lift is real. Those are healthy counts. If the same calculation had produced 25 conversions per arm, the normal approximation would be doing more work than the data can support.
How to read the number, and what to do when it is impossible
Start with the duration, not the sample. Convert visitors into days at your real bucketing rate — not sessions, not pageviews, but the units you randomise. Then round up to a whole number of weeks. Traffic composition swings hard between Monday and Saturday, and a test that runs Tuesday to Friday has measured a Tuesday-to-Friday audience. One full week is the floor; two is the usual professional minimum because it lets you compare week one against week two and see whether the effect is stable or a novelty spike.
Next, check the ceiling. Tests that run past about four to six weeks start to decay: cookies are cleared, people switch devices, promotions launch, and the two arms stop being the clean random split you started with. If your calculation demands ten weeks, the honest options are to test a bolder change, test a metric earlier in the funnel, pool several low-traffic pages into one test, or raise your MDE and accept that you will only ever detect large effects.
Finally, sanity-check the conversions. Roughly 100 conversions per arm is a practical floor for the normal approximation; a few hundred is comfortable. If your traffic gives you 12 conversions a week, no amount of clever statistics fixes it — switch to a funnel diagnosis and qualitative research, and reserve experimentation for the pages that actually get traffic.
When the test finishes, do not eyeball the difference. Feed the four counts into the A/B test significance calculator, and read the confidence interval rather than just the verdict: an interval that spans zero and also spans a lift you would happily ship means “inconclusive”, not “no effect”.
Visitors per variation at 95% confidence and 80% power
| Baseline rate | Detect +5% | Detect +10% | Detect +20% | Detect +50% |
|---|---|---|---|---|
| 0.5% | 1,280,619 | 327,922 | 85,862 | 15,599 |
| 1% | 637,010 | 163,095 | 42,693 | 7,750 |
| 2% | 315,206 | 80,682 | 21,109 | 3,826 |
| 5% | 122,124 | 31,234 | 8,158 | 1,471 |
| 10% | 57,763 | 14,751 | 3,841 | 686 |
| 20% | 25,583 | 6,510 | 1,683 | 294 |
| 30% | 14,856 | 3,763 | 963 | 163 |
Computed from the formula above and rounded up. A 1% baseline needs roughly eleven times the traffic of a 10% baseline for the same relative lift.
What alpha and power actually promise
α = 0.05 means: if the variant is genuinely identical to control, you will still declare a winner 5% of the time. Run twenty null experiments and expect one false champion. Power = 80% means: if the variant genuinely differs by exactly your MDE, you will detect it 80% of the time — and miss it the other 20%. Neither number is a probability that your specific result is correct, and neither is a statement about effects smaller or larger than the MDE, which are detected less and more often respectively.
These are the conventions of the Neyman–Pearson framework, and 0.05 and 80% are conventions, not laws. A change that is cheap to ship and cheap to reverse can justify α = 0.10; a change to pricing or a regulated flow deserves α = 0.01.
Mistakes that wreck a sample-size plan
- Confusing relative and absolute effects. On a 5% baseline, “1% lift” might mean 5.05% or 6%. Those two plans differ by a factor of roughly 370 in required traffic — about 3.0 million visitors per arm against 8,158. State which you mean, every time.
- Sizing on total traffic instead of per variation. The formula returns n per arm. A two-arm test needs 2n; a four-arm test needs 4n and a stricter alpha on top.
- Forgetting multiple comparisons. Testing three variants against control is three chances to get lucky. Without a correction your real false-positive rate is nearer 14% than 5%.
- Randomising sessions but sizing on visitors. Pick one unit — usually the user — and use it consistently for randomisation, sample size and analysis. Mixing them breaks the independence the formula assumes.
- Stopping early because the result looks good. The single most common way a well-designed test produces a wrong answer. If you need to stop early, use a sequential or always-valid method designed for it, not a fixed-horizon test read early.
- Sizing for one metric, then reading fifteen. Every extra metric you scan is another lottery ticket. Nominate one primary metric up front; treat the rest as diagnostics.
- Using this formula for revenue per visitor. Revenue is heavy-tailed, not binary. A proportion formula badly understates the sample a revenue metric needs — size that with the observed variance instead.
- Ignoring the traffic split. A 90/10 split does not need n in each arm; the small arm becomes the binding constraint and the total climbs sharply. Keep splits even unless you have a reason not to.
What this calculator assumes, and when to use something else
The expression above is the two-proportion normal approximation given by Fleiss, Levin and Paik in Statistical Methods for Rates and Proportions, taken without the continuity correction; it is the same relation R evaluates in power.prop.test with strict = FALSE, which is why the two agree to the visitor. Adding Fleiss's continuity correction raises the requirement by a few percent and is the conservative choice when counts are small.
The math assumes independent observations, a fixed sample size decided in advance, equal-sized arms, a binary outcome per unit, and a normal approximation to the binomial. That covers the overwhelming majority of web conversion tests. It does not cover several cases you will eventually meet.
Very small counts. Below roughly five expected conversions per arm the normal approximation is unreliable and Fisher's exact test or an exact binomial power calculation is the right tool.
Continuous or heavy-tailed metrics. Revenue per visitor, average order value and session duration need a variance-based sample size, because a handful of large orders dominates the variance. Estimate the standard deviation from your own data first, then size for a difference in means. If revenue is what you actually care about, pair this with the revenue per visitor calculator so you know what a percentage lift is worth.
Clustered or non-independent units. If you randomise by account and measure by seat, or run a marketplace test where buyers and sellers interact, observations are correlated and the effective sample is smaller than the raw count. Cluster-randomised designs need a design-effect inflation factor.
Sequential and Bayesian designs. Group-sequential boundaries and always-valid p-values are built for continuous monitoring and can stop early legitimately, at the cost of a somewhat larger maximum sample. Bayesian designs answer a different question — the probability the variant is better — and need a decision rule and a prior rather than a sample size.
One practical note on interference: if you also need to hit a traffic target for the test to be worth running at all, model the traffic side separately with the website traffic goal calculator, and price the winner's value with customer lifetime value rather than first-order conversions alone.
Key terms
- Minimum detectable effect (MDE)
- The smallest true difference the test is designed to find. Smaller MDE, larger sample — by the square of the ratio.
- Significance level (α)
- The false-positive rate you accept: the probability of declaring a winner when the two arms are identical.
- Power (1 − β)
- The probability of detecting an effect of exactly the MDE size when it is real. Conventionally 80%.
- Two-sided test
- A test that can detect a change in either direction. It splits α between the two tails, which is why it needs more data than a one-sided test.
- Bonferroni correction
- Dividing α by the number of comparisons to keep the family-wide false-positive rate at the nominal level. Conservative but simple.
- Fixed-horizon test
- A test whose sample size is decided before launch and read only once, at the end. The assumption behind every classical p-value.
- Sample ratio mismatch (SRM)
- When the observed split between arms differs from the intended split by more than chance allows — a signal that the experiment's plumbing, not the variant, produced the result.