How to decide whether an A/B test is worth running

The question, before the test

Whether a test is worth running is a question with a numerical answer, and it is answerable before anyone builds anything. It takes four inputs: the baseline rate of the metric at the step under test, the smallest effect the team would act on, the traffic that reaches the step, and how long the team is willing to wait. From the first two comes the sample the test needs. From the third comes how many days that sample takes. The fourth says whether that is acceptable. A test whose days come out at a quarter is not a test, whatever the idea behind it; it is a decision that has been postponed to a date on which nobody will know more.

The experimentation platforms say the same thing in their own words. Statsig's power analysis estimates the smallest change an experiment can reliably detect, and the duration and allocation it needs, from the traffic and the metric's variance, before the experiment starts [3]. PostHog computes the running time up front from the sample size and the visitors a day, and advises starting from a minimum detectable effect of about thirty percent when unsure and adjusting to what the sample allows [4]. The sample size calculator in this library does the arithmetic and prints the band. The arithmetic is not the hard part. The hard part is doing it before the test rather than after, and believing the answer when it says no.

The four inputs

The baseline. The rate today, on the population the test will run on, measured over a period that covers a full weekly cycle. A baseline read from a good week or from a wider population than the test will see puts every later number off.

The minimum detectable effect. Not the effect the team hopes for, and not the effect the change might have, but the smallest change that would make the team ship the variant. A five percent relative lift on paid activation might be worth shipping; a half percent might not be worth the complexity it adds. This number is a business decision dressed as a statistical one, and it is the input most teams get wrong, because they set it to what they expect rather than to what they would act on.

The traffic at the step. Eligible visitors a day: the ones who reach the screen under test and meet the cohort's conditions. Not site visitors. A pricing page sees a fraction of the signup screen's traffic, and only the signed-in free-plan visitors of it can upgrade, which is why the pricing experiment in this library warns that its own sample is the slowest to reach.

The patience. How long the test may run. The answer is bounded on both sides. Kohavi and colleagues recommend at least a week or two, then multiples of a week, so that the weekly cycle and its primacy and newness effects are covered [2]. And they give a worked case where a test would need over 125 days, a period they consider too long for a reliable result, because over that span cookie churn and other drift start contaminating the data [2].

The arithmetic, on an example

The sample size guide works through the formula. Its reference case is the one to keep in mind: a baseline of five percent, a relative minimum detectable effect of ten percent, a two-sided significance level of five percent and eighty percent power give about 31,200 visitors per variant, and so about 62,500 in all at an even split. The same authors' rule of thumb compresses the whole calculation to sixteen times the variance over the square of the effect, at the same conventional confidence and power [2].

Now put that against the front page's example funnel, which is the blueprint's example and not a customer's: 10,000 visits a month, 4,400 signups, 3,900 reaching workspace setup, 1,900 with a first project, 1,400 paying. A test at the setup step with a five percent baseline and a ten percent relative effect on 3,900 eligible visitors a month needs sixteen months. Not sixteen days. That test is not feasible on that traffic at that effect, and no amount of wanting the answer changes it. It becomes feasible by testing a bigger change, by choosing a metric closer to the change with a higher baseline, by widening the cohort, or by not running it and deciding the change on judgment.

The signup screen on the same funnel sees 10,000 visitors a month, with a baseline of forty four percent. A high baseline and more traffic together bring the same ten percent relative effect to a sample of a few thousand a variant and a couple of weeks. That is the whole reason signup tests resolve fastest, and it is why the signup experiment is the one on this funnel a team could run next week.

Three bands

Read the days and place the test in a band. Under fourteen days: feasible at this traffic, and still worth running for a full week or two so the weekly cycle is covered. Fourteen to fifty six days: long. Plan for seasonality, decide the sample in advance, and do not peek. Over fifty six days: the test is unlikely to reach reliable evidence in a reasonable time at this traffic. Consider a larger change, a metric closer to the change, a bigger cohort, or not running the test at all.

The bands are Athren's own reading of the published guidance on cycles and drift, not a standard, and the boundaries are round on purpose. Two weeks is the shortest run that covers a weekly cycle with room to spare; eight weeks is where seasonality, cookie churn and the team's own patience start to cost more than the answer is worth.

Why not just run it and see

Because a test read whenever it looks good does not produce the evidence it claims to. Repeated significance testing always increases the rate of false positives: a test declared at the first moment it shows significance, at a nominal five percent, is wrong far more often than five percent of the time [1]. The remedy Evan Miller gives is the one this whole guide rests on: decide on a sample size in advance and wait until the experiment is over [1]. A test that cannot reach its sample in a reasonable time is a test that will be read early, and read early it is a coin toss with a chart.

There is a second cost. A test running for a quarter occupies the screen it runs on, the flag it runs behind and the attention of the people watching it. Every other experiment on that step waits. The throughput guide makes the case that the number of experiments a team finishes is what its programme returns; a test that cannot finish lowers that number for as long as it runs.

When not to test, plainly

Athren's position is that knowing when not to test is part of experimentation, not a failure of nerve. Some changes should be decided and shipped: the arithmetic says the test cannot finish, the change is small and reversible, and the cost of being wrong is lower than the cost of a quarter spent finding out. Some changes should not be tested because they cannot be reverted for the people they touched, which is why price changes are not experiments in this library. And some changes should be tested at a different step, where the traffic is, against a metric closer to the change.

Decide first whether a test can reach an answer. Then decide whether the answer is worth what it costs. Only then decide to run it. Every experiment in this library carries that reasoning in its traffic plan, and the ones that would not be feasible on a small product say so.