A/B test sample size explained

What the sample size is for

An A/B test compares the rate of something, a signup, an activation, an upgrade, between two groups of visitors who differ only in which version they saw. The rates in the two groups will differ by chance even when the versions are the same, and the question a test answers is whether the difference it observed is larger than chance would produce. The sample size is the number of visitors each group needs before that question can be answered with the reliability the team decided on in advance. Too few, and a real effect hides in the noise or a chance difference reads as a win. The number is not a formality; it is the thing that makes the result mean anything.

The four inputs

The baseline conversion rate. The rate in the control group, which is the rate today. Call it p1. At a five percent baseline, one visitor in twenty converts.

The minimum detectable effect. The smallest change from the baseline the test should be able to find. It can be stated relative to the baseline, ten percent of five percent is half a percentage point, or absolute, half a percentage point directly. Either way it fixes the rate the variant would have if the effect is real, p2. Evan Miller's calculator defines it as the smallest effect that will be detected the stated share of the time [3], and the choice is the team's: not the effect they hope for but the smallest one they would ship.

The significance level. The chance the test is willing to take of declaring an effect when there is none, written as alpha. Five percent is the convention, and two-sided, so that a variant that does worse is caught as well as one that does better. Evan Miller's phrasing is exact: the percent of the time a difference will be detected, assuming one does not exist [3].

The power. The chance that a real effect of the stated size is detected, written as one minus beta, where beta is the chance of missing it [2]. Eighty percent is the convention; ninety costs more visitors. Evan Miller: the percent of the time the minimum effect size will be detected, assuming it exists [3].

The formula

The test behind the arithmetic is the two-sample test of proportions, two-sided. With p1 the baseline, p2 the variant rate, k the ratio of variant to control allocation, and p̄ the allocation-weighted average of the two rates, the control group's sample is

n_control = ( z_α · √((1 + 1/k) · p̄ · (1 − p̄)) + z_β · √(p1·(1 − p1) + p2·(1 − p2) / k) )² / (p2 − p1)²
n_variant = k · n_control

where z_α is the standard normal quantile at one minus alpha over two, about 1.96 at five percent two-sided, and z_β is the quantile at the power, about 0.84 at eighty percent. At an even split, k is one and the first square root becomes the familiar √(2 · p̄ · (1 − p̄)). The NIST handbook gives the form for a proportion against a baseline, with the two z values and the two binomial variances in exactly this arrangement [1]; the two-sample version carries both groups' variances, which is what the second square root holds.

The structure is what to remember. The numerator is the noise: how much two proportions wobble by chance, scaled by how sure the team wants to be. The denominator is the signal: the effect, squared. Halving the effect quadruples the sample, which follows directly from the effect being squared in the denominator [2], and Kohavi and colleagues state the same law in its dramatic form: an effect ten times smaller needs a hundred times more users [4].

The worked example

Take the reference case: a baseline of five percent, a relative minimum detectable effect of ten percent, so p2 is five and a half percent, alpha five percent two-sided, eighty percent power, an even split.

• p̄ is 0.0525.
• The first square root: √(2 · 0.0525 · 0.9475) is about 0.3154; times 1.96 is about 0.618.
• The second: √(0.05 · 0.95 + 0.055 · 0.945) is about 0.3154; times 0.8416 is about 0.266.
• The sum squared is about 0.781. The effect squared is 0.005², or 0.000025.

Dividing gives about 31,234 visitors per variant, so about 62,500 in all. The A/B test sample size calculator in this library runs this formula and this case, and rounds up to the next whole visitor, because a fraction of a visitor is not a sample.

Change one input and watch the sample move. A twenty percent relative effect, p2 of six percent, needs about a quarter of the visitors. A one percent baseline with the same ten percent relative effect needs about five times as many, because a rare event has less signal per visitor. Ninety percent power adds about a third.

A rule of thumb

For a quick read without the full formula, Kohavi and colleagues give a rule that fits on a line: the number of users in each variant is about sixteen times the variance of the metric divided by the square of the change to detect, at the conventional ninety five percent confidence and eighty percent power [4]. For a rate, the variance is p · (1 − p). At a five percent baseline and a half-point effect, sixteen times 0.0475 over 0.000025 is about 30,400 a variant, within a few percent of the exact figure. The rule is good enough to decide whether a test is in the right order of magnitude before anyone opens a calculator, which is the decision the feasibility guide is about.

What the number assumes

The formula assumes visitors are independent of each other, that the metric is binary, that the test runs to a fixed sample and is read once at the end, and that the variant's rate is the stated one if the effect is real. Each assumption can fail. Visitors who return count twice unless the unit of assignment is the visitor. A metric that is a count or an amount, revenue per visitor, has a variance the binomial formula does not know. Reading the test early, or repeatedly, changes the false positive rate the significance level promised, and the feasibility guide says how badly. Sequential designs and variance reduction methods change the arithmetic, and a sample computed this way does not apply to them.

None of that makes the number wrong for what it is: the sample a fixed-horizon test of a rate needs, read once. It is the number every experiment in this library sizes against, and the onboarding experiment shows where it goes in a traffic plan.