A team has a change it wants to test, a conversion rate it wants to improve, and a limited amount of traffic. The question comes quickly: do we have enough visitors to run an A/B test?
There is no single traffic threshold. The answer depends on the current conversion rate, the size of improvement you need to detect, and how the experiment is designed. A test also needs to produce an answer while there is still time to use it.
A website may have enough traffic to detect a substantial improvement but need months to distinguish a modest gain from ordinary variation. That difference can change which ideas are worth testing.
What determines the sample size
For a conventional A/B test comparing two conversion rates, four inputs shape the calculation.
| Input | What it means | Why it matters |
|---|---|---|
| Baseline conversion rate | The current conversion rate for the eligible audience | At the same relative improvement, a low baseline generally requires more visitors |
| Minimum detectable effect | The effect size the test is designed to detect with the chosen power | Smaller effects require larger samples |
| Significance level | The tolerated false-positive rate when there is no real difference, under the test assumptions | A stricter threshold requires more evidence |
| Statistical power | The probability of detecting the specified effect when it exists | Higher power requires a larger sample |
A sample-size calculation makes their trade-offs explicit. NIST explains the relationship between effect size, significance level and power.
An 80% power setting means that, across repeated equivalent experiments where the specified effect exists, around 80% would detect it. It does not mean a reported winner has an 80% probability of being correct.
Nor does reaching the planned sample guarantee a significant result. The actual effect may be smaller than the one the test was designed to detect, or there may be no effect. Sample-size planning establishes the sensitivity of the experiment. Research on sample-size calculations for A/B testing.
The improvement you want to detect changes the timeline
Consider a hypothetical website with a 2% conversion rate and 500 new eligible visitors entering an experiment each day. Visitors are divided equally between the existing experience and one alternative.
The following estimates use a conventional fixed-sample test, a two-sided 5% significance threshold and 80% power.
| Relative improvement to detect | Conversion-rate change | Approximate visitors across both groups | Days to collect the sample |
|---|---|---|---|
| 10% | 2% to 2.2% | 161,400 | 323 |
| 20% | 2% to 2.4% | 42,200 | 85 |
| 50% | 2% to 3% | 7,700 | 16 |
The estimates assume independent visitors, a converted-or-not outcome for each visitor, and steady enrolment. Visitor totals are rounded; durations use the unrounded samples. Allow additional time for delayed conversions and relevant business cycles. These calculations use a normal approximation for two independent proportions, so other statistical methods may give different estimates. Statistical method reference.
Moving from 2% to 2.2% is a 10% relative improvement and a 0.2 percentage-point increase. Check whether your calculator asks for absolute or relative lift. Entering one as the other can give a very different sample requirement.
At 500 eligible visitors a day, the 50% improvement takes about two weeks to investigate. The 10% improvement takes most of a year. The traffic has not changed.
Selecting a 50% improvement in a calculator makes the sample smaller. It does not make that improvement more plausible. A test designed around that effect has less power to detect smaller gains that might still matter commercially.
Decide what improvement would be worth acting on
Before choosing the minimum detectable effect, establish what a worthwhile improvement would mean for the business.
Consider the cost of implementation and maintenance, the value of each additional outcome, and the risks associated with the change. An inexpensive improvement repeated across a large audience may justify attention at a modest uplift. A costly integration may need a much stronger commercial case.
This is practical significance: whether an improvement is worth acting on. A statistically significant result still needs to justify its implementation cost and any trade-offs elsewhere in the journey.
The smallest worthwhile improvement may be below what your available sample can reliably detect. If a 10% improvement would be valuable but the available sample only supports reliable detection of a much larger effect, a non-significant result will not resolve whether the smaller gain exists.
Look at the range of effects consistent with the result. If the interval still includes meaningful benefit and meaningful harm, the experiment has left an important question unanswered.
Count the audience that can enter the experiment
Use the eligible audience when estimating duration. A checkout experiment cannot draw on every person who visits the homepage. Device restrictions, targeting rules and the share of traffic allocated to the experiment further affect enrolment.
Keep the baseline and traffic estimate aligned with the metric and population being tested. A sitewide session conversion rate may be unsuitable for an experiment measuring whether eligible users convert.
Repeated sessions from the same person are not automatically independent observations. The calculation and analysis need to reflect how users are assigned and measured. Research on correlated observations in online experiments.
Check that measurement works before starting. If the conversion event is duplicated or records an attempted action as a completed one, collecting more observations will not repair the definition. Our CRO audit guide explains how measurement checks fit into the wider diagnosis.
A long test needs a reason to run
A three-month test may be worthwhile for a decision that will affect customers for years. It is harder to justify for a campaign that ends next month.
Will the result still be useful then? Can the experience remain sufficiently stable? Will a seasonal promotion dominate the audience? How long do conversions take to mature? What other experiments or delivery work would be delayed?
The cost of waiting belongs in the decision alongside the cost of getting it wrong.
A short experiment brief should cover:
- The customer problem and the decision the result will inform.
- The primary outcome and measures that must not deteriorate.
- The eligible audience, baseline and worthwhile effect sizes.
- The statistical method, estimated sample, duration and stopping rules.
- The action you would take after a positive, negative or inconclusive result.
If the team cannot explain how the outcome would change a decision, revisit the experiment brief.
Use the stopping rules that match the method
For a conventional fixed-sample test, repeatedly checking the result and stopping at the first significant reading increases false-positive risk. Looking at a dashboard for operational problems is different from repeatedly using it to declare a winner.
Properly designed sequential methods allow ongoing evaluation with appropriate error control. Bayesian methods use different assumptions and decision rules. Match planning and interpretation to the method configured in the platform; a generic calculator may not reproduce its results. Optimizely explains the differences between statistical approaches.
Changing the significance threshold after seeing an inconvenient result changes the standard of evidence. It does not add information. Likewise, an easier-to-measure click cannot establish a revenue benefit simply because it reaches significance sooner.
When an A/B test is impractical
If the sample requirement puts a useful result out of reach, return to the customer problem.
Fix a reproducible functional fault and verify that the affected task works. Use interviews or usability research to investigate what people cannot understand or complete. For a change supported by evidence, assess whether a limited release with monitoring and a rollback plan is proportionate to the risk.
The conclusion should match the evidence. Usability research can reveal a task failure. A successful repair can demonstrate that the task now works. A rise in conversion after release may have several explanations and does not, by itself, establish the change’s causal effect.
At Foster & Funnel, our Growth & Optimisation work brings together experimentation, journey analysis and performance diagnostics. If traffic is limiting what you can test, start by establishing which decision needs evidence and what it would be worth to resolve it.
