Experiment statistics
How many visitors a test needs, what power and minimum detectable effect mean, and how to tell before it starts whether a test can reach an answer.
What this category answers
The questions a growth product manager or an experimentation lead has to answer without a statistician on call: how large a sample a test needs and why, what a minimum detectable effect is and how to choose one, what significance and power protect against, what stopping early costs, and how to decide, from traffic and baseline alone, whether a test is worth running or should be a decision instead. Written for someone who owns the test, with the formulas shown and the assumptions named, and the sample size calculator beside them.
Experiment statistics guides
- Experiment statisticsHow to decide whether an A/B test is worth runningCompute the sample from the baseline and the smallest effect worth acting on, divide by the traffic at the step, and read the days. Under two weeks, run it; over eight, decide the change by judgment instead.
- Experiment statisticsA/B test sample size explainedThe sample an A/B test needs follows from the baseline rate, the smallest effect worth detecting, the significance level and the power. At a five percent baseline and a ten percent lift, that is about 31,200 a variant.
Give us one funnel.
Twenty five minutes, about how you run experiments today. No access, no commitment.