How Many Visitors Does an A/B Test Need?

Four inputs, and you choose all of them

A sample size is not discovered, it is specified. Four numbers go in. The baseline conversion rate is the one you already have — measure it over a full number of weeks so weekday and weekend traffic are both represented. The minimum detectable effect is the smallest improvement you would actually act on, and it is normally stated as a relative lift: a 10% lift on a 3% baseline means a variant rate of 3.3%. The significance level α is your tolerance for declaring a winner that does not exist, conventionally 5% two-sided. And the power is your tolerance for missing a real winner, conventionally 80%, which means you accept a one-in-five chance of failing to detect an effect that is genuinely there.

The one people get wrong is the minimum detectable effect, because it feels like an output rather than an input. It is not. It is a commitment made in advance: you are declaring that a lift smaller than this is not worth the traffic it would take to prove. Choosing it honestly is the single most useful thing you can do for a testing programme, because choosing it optimistically produces tests that are too short to conclude anything and choosing it timidly produces tests nobody has the traffic to finish.

The formula, with every symbol named

For two independent proportions with equal group sizes, the standard normal-approximation result is n per variant = [z(α/2) × √(2 p̄ (1 − p̄)) + z(β) × √(p1(1 − p1) + p2(1 − p2))]² ÷ (p2 − p1)². Here p1 is the baseline rate, p2 is the variant rate you want to be able to detect, and p̄ is their average, (p1 + p2) ÷ 2. The term z(α/2) is the standard normal value leaving α/2 in each tail: 1.96 for a two-sided test at α = 0.05. The term z(β) is the value leaving β in one tail, where β is one minus the power: 0.84 for 80% power, 1.28 for 90%.

Two structural facts fall straight out of the algebra and are worth more than the formula itself. First, the difference p2 − p1 appears squared in the denominator, so the required traffic scales as one over the square of the effect you are chasing. Halve the effect and you roughly quadruple the traffic. Second, moving from 80% to 90% power multiplies the requirement by a fixed factor of about 1.34 regardless of the baseline or the lift, because the only thing that changed is z(β) going from 0.84 to 1.28. Extra confidence is expensive but predictably so; a smaller target effect is expensive in a way that runs away from you.

How fast it explodes, at a 3% baseline

Run the formula at a 3% baseline and the shape of the problem becomes obvious. To detect a 50% relative lift — 3% to 4.5%, an absolute gap of 1.50 percentage points — you need 2,518 visitors per variant at 80% power, so 5,036 in total. Half of two days of traffic at 10,000 a week. For a 25% lift, 3% to 3.75%, the gap falls to 0.75 points and the requirement jumps to 9,100 per variant, 18,200 in total, about 1.8 weeks. For a 10% lift, 3% to 3.3%, a gap of 0.30 points, you need 53,211 per variant, 106,422 in total, over ten weeks. And for a 5% lift, 3% to 3.15%, a gap of 0.15 points, you need 207,938 per variant, 415,876 in total, more than 41 weeks.

The baseline matters as much as the lift, and in the same direction: low-converting pages are expensive to test. A 10% relative lift needs 53,211 visitors per variant at a 3% baseline, but only 14,751 at a 10% baseline and 3,763 at a 30% baseline — while at a 1% baseline it needs 163,095. Rare events carry more relative noise, so you have to watch longer to see through it. This is why checkout-step tests, which run on high-converting pages, conclude in days while homepage-to-purchase tests on the same site never conclude at all.

Relative or absolute? The mistake that shrinks a test 463-fold

Almost every sample-size tool asks for the effect in one of two forms and does not always say which. A 10% relative lift on a 3% baseline means moving to 3.3%. A 10 percentage point improvement on the same baseline means moving to 13%. These are not close: the first needs 53,211 visitors per variant, the second needs 115. Enter the wrong one and your test is 463 times too small, will reach its planned sample size in an afternoon, and will report whatever noise happened to be in the data that day.

The habit that prevents it costs nothing. Write down both forms before you touch the calculator: the baseline rate, the variant rate, and the absolute gap in percentage points. If you cannot state the variant rate as a number, you have not specified an effect. Doing this also disciplines the conversation with stakeholders, because a request for a 10% lift sounds modest right up to the moment someone writes down that it means 3% becoming 3.3% and that proving it takes ten weeks of the site's entire traffic.

Peeking: why "it went significant" is not a result

The 5% false-positive rate that significance testing promises applies to one test, run once, on a sample whose size was fixed in advance. Watch the dashboard every day and stop the moment the p-value dips below 0.05 and you are not running that test any more. You are running a procedure that gets many chances to cross the line, and its error rate is the probability of crossing at least once — which is much larger.

The size of the inflation is measurable, and we measured it: a simulated A/A test — both arms at exactly 3%, no real difference whatsoever, 9,100 visitors per arm, 20,000 repetitions — declared a winner 5.0% of the time when the data was examined once at the end. Examined twice, it declared a winner 8.3% of the time. At five equally spaced looks, 14.6%. At ten, 19.5%. At twenty, 24.6%. Those figures reproduce the profile Armitage, McPherson and Rowe published in 1969, and the pattern continues: with continuous monitoring and no stopping rule, the probability of eventually crossing the line approaches certainty even when both variants are identical. One in five of the wins celebrated on a dashboard checked daily for a fortnight is nothing at all.

What to do instead

The simplest fix is to fix the horizon and honour it. Compute the sample size before launch, convert it into a number of whole weeks so every day of the week is represented equally, write that end date down, and read the result once — at the end. You may look at the dashboard in the meantime to check the test is technically healthy, that traffic is splitting evenly and nothing has broken. You may not stop on what you see there. The difference between monitoring and deciding is the entire safeguard.

If you genuinely need the option to stop early — because the variant might be harming revenue, or because the traffic cost of a full run is prohibitive — then use a method built for it rather than breaking the one you have. Group sequential designs, of which Pocock's 1977 boundaries are the classic example, let you look a pre-declared number of times and spend your error budget across those looks by tightening the threshold at each one. Sequential and Bayesian approaches offered by several testing platforms achieve the same thing with continuously valid bounds. All of them cost you something: a stricter threshold, or a larger maximum sample, or both. That cost is the honest price of the flexibility, and it is far cheaper than a programme that ships a quarter of its losers as winners.

Frequently asked questions

My site does not have enough traffic. What can I test?

Bigger changes, and events that happen more often. The two levers in the formula that you actually control are the size of the effect and the baseline rate. Testing a full redesign of a page instead of a button colour raises the plausible effect from a few percent to tens of percent, and a 50% relative lift on a 3% baseline needs only 2,518 visitors per variant rather than the 207,938 a 5% lift demands. Moving the measured event further up the funnel does the same job: a test measured on add-to-cart, which might convert at 10%, needs 14,751 visitors per variant for a 10% lift, against 53,211 for the same lift measured on a 3% purchase rate. If neither is available, run the change without a test and judge it on a before-and-after basis, but call that what it is — a decision, not evidence.

Should I use 80% or 90% power?

Eighty percent is the default, and the upgrade costs a fixed 34% more traffic in every case: at a 3% baseline and a 25% lift the requirement moves from 9,100 to 12,182 visitors per variant, and at a 10% lift from 53,211 to 71,233. The ratio never changes because only z(β) moved, from 0.84 to 1.28. Buy the extra power when a false negative is expensive — when the variant took a quarter of engineering's time to build, when the decision blocks a roadmap, or when you will not get another chance to run the test this year. Stay at 80% when the change is cheap to revisit, because the traffic is better spent on a second test than on a third decimal place of certainty in the first.

Can I stop the test early if one variant is clearly winning?

Not on a fixed-horizon design, no — that is exactly the move the design cannot absorb. A simulated A/A test with no real difference between the arms produced a significant result 5.0% of the time when read once, 14.6% at five looks and 19.5% at ten. An early lead is also the least trustworthy kind of lead, because differences are noisiest when the sample is smallest, and a large apparent effect in week one is more likely to be a small sample than a large effect. If early stopping is a genuine requirement rather than impatience, adopt a group sequential design with declared interim looks and adjusted thresholds, or a testing platform that publishes continuously valid bounds, and accept the larger maximum sample that comes with it.

Does the sample size mean visitors or conversions?

Visitors — or more precisely, the units you randomised, each counted once. The figures in the table are visitors per variant, so 9,100 for a 25% lift means 9,100 in the control and 9,100 in the variant, 18,200 in total. Randomise on the visitor, not the session, or someone who returns three times gets counted three times and their behaviour is correlated across those visits, which quietly shrinks your true sample below the number you are watching. Count each visitor's outcome once as well: a visitor either converted during the test window or did not.

Why should a test run for whole weeks?

Because conversion rates have a weekly rhythm and a test that stops mid-week samples that rhythm unevenly. Weekday and weekend visitors differ in intent, device and how much time they have; so do the days around a payday or a weekly newsletter send. A test that runs eleven days gives one weekend more weight in the arithmetic than the other, which shows up as a difference between arms only by luck, but shows up as instability in the result reliably. Round the computed sample size up to the next whole number of weeks, start and end on the same weekday, and the seasonality cancels between the two arms instead of leaking into the estimate.