Quick Summary
- Run a sample size calculation BEFORE launching any A/B test, not after.
- The three inputs you need: baseline conversion rate, minimum detectable effect (MDE), and desired statistical power.
- Peeking at results early and stopping when you see significance inflates false positive rates to 40%+ (simulation data, 2025).
- Low-traffic sites need larger MDEs or longer run times, there is no shortcut around math.
- Sequential testing methods let you peek safely, but they require more total sample to compensate.
What It Actually Is
Sample size math answers one question: how many visitors do you need in each variant before your A/B test result can be trusted?
Think of it like flipping a coin. If you flip it 10 times and get 6 heads, you cannot conclude the coin is biased. Flip it 1,000 times and get 600 heads, now you have real evidence. A/B testing works the same way. You need enough flips (visitors) before the difference between variants stops being noise.
The formula depends on three things:
- Baseline conversion rate, what your control currently converts at.
- Minimum detectable effect (MDE), the smallest improvement worth acting on.
- Statistical power, how confident you want to be that you will detect a real effect if one exists (typically 80%).
A fourth input, statistical significance threshold (alpha), is usually set at 0.05 (95% confidence).
Why It Matters
Running tests without a sample size calculation is one of the most common and costly CRO mistakes:
- False positives from peeking: A 2025 simulation study by Alex Molas found that checking results 10 or more times during a test and stopping at the first significant result inflates false positive rates above 40%, compared to the nominal 5% alpha level. Source: Alex Molas, October 2025
- Industry scale: Booking.com runs more than 25,000 experiments per year. At that volume, even a 1% false positive rate means 250 bad decisions annually. They publish their methodology partly because the cost of incorrect conclusions at scale is enormous.
- Low-frequency conversions break standard tests: Airbnb's engineering team has published research showing that rare events (purchases, sign-ups for premium tiers) require so many visitors that standard A/B testing becomes impractical. They use interleaving and other techniques specifically because sample size requirements become prohibitive.
- Most tests do not win: Microsoft's ExP platform, which runs over 10,000 tests per year across Microsoft products, reports that only 1 in 3 tests produces a positive result. Running underpowered tests makes that ratio worse because you cannot reliably detect the real winners.
- Industry benchmarks: AB Tasty recommends a minimum of 30,000 visitors and 3,000 conversions as a starting point for reliable test results. Most small sites cannot reach this threshold in a reasonable timeframe without widening their MDE.
The 5-Step Playbook
Step 1, Measure your baseline conversion rate
Pull data from the last 30 to 90 days for the specific page or funnel step you are testing. Do not use site-wide averages. If you are testing a product page checkout button, use the conversion rate for that page only.
Example: 1,200 conversions from 40,000 visitors = 3.0% baseline conversion rate.
Step 2, Choose your MDE
The MDE is the smallest relative improvement you care about. Smaller MDEs require larger samples. Use this table as a starting point:
| Monthly Traffic | Realistic MDE |
|---|---|
| Under 10,000 visitors | 20%+ relative improvement |
| 10,000 to 50,000 visitors | 10 to 20% relative improvement |
| 50,000 to 200,000 visitors | 5 to 10% relative improvement |
| Over 200,000 visitors | 2 to 5% relative improvement |
A 10% relative MDE on a 3% baseline means you are trying to detect a move from 3.0% to 3.3%.
Step 3, Run the calculation
Use an online calculator (Evan Miller's is the standard) or the formula below:
n = 16 * sigma^2 / delta^2
Where sigma is the standard deviation of your metric and delta is the absolute difference you want to detect. For proportions, most calculators handle this automatically.
Example: baseline 3%, MDE 10% relative (target 3.3%), alpha 0.05, power 80% gives approximately 14,000 visitors per variant, or 28,000 total.
Step 4, Estimate runtime
Divide your required sample by your daily traffic to the test page.
Example: 28,000 total visitors needed, 2,000 daily visitors to the page = 14 days minimum runtime.
Never stop a test early because it reached significance before the planned runtime. Peeking and stopping early is how you get 40%+ false positive rates. Set your end date before you launch.
Step 5, Validate results after the test
When the test ends, check:
- Did you hit your planned sample size?
- Did you run for at least one full business cycle (usually 7 days minimum, two weeks preferred)?
- Is the result above your significance threshold AND above your MDE?
Only then should you call a winner.
Decision Flowchart
Real Company Examples
Booking.com, scaling sample size math to 25,000+ tests per year
Booking.com's experimentation team has written publicly about their statistical infrastructure. At 25,000+ tests per year with hundreds of metrics per test, they cannot manually set sample sizes for each experiment. They use automated power analysis that pulls baseline metrics from their data warehouse and sets runtime automatically. Every test gets a planned end date before launch. The team explicitly trains new analysts that peeking and stopping early is a fireable-level mistake because of how badly it corrupts their decision database over time.
Airbnb, when standard A/B testing breaks down
Airbnb's engineering blog describes their challenge with low-frequency conversion events like first bookings and host sign-ups. A 1% MDE on a 0.5% conversion rate requires millions of visitors per variant, an impractical runtime. Their solution was to move to interleaving experiments (showing multiple variants to the same user in sequence) and Bayesian sequential testing for these rare events. Standard fixed-horizon sample size math still governs their high-volume experiments (search ranking, page layout), but they built separate infrastructure for the long-tail metrics.
Common Mistakes
Mistake 1, Calculating sample size after the test ends. Some teams run tests until they see a result they like and then calculate whether the sample was sufficient. This is backwards and produces whatever answer you want to hear.
Mistake 2, Using site-wide traffic instead of page traffic. If you get 100,000 monthly visits but only 5,000 reach your test page, your effective traffic is 5,000. Use the number that actually sees the test.
Mistake 3, Ignoring multiple variants. Running an A/B/C/D test with four variants does not give you the same power as a two-variant test at the same traffic. You need to adjust your sample size upward or your significance threshold downward (Bonferroni correction).
Mistake 4, Setting MDE too small for your traffic. Setting a 1% relative MDE because you think it should be detectable sounds rigorous. In practice it requires 10x more traffic than a 10% MDE. If you cannot reach that sample in 8 weeks, you will never run a conclusive test.
Mistake 5, Forgetting novelty effect. The first few days of a test often show inflated engagement just because something is new. This is one reason two-week minimums matter even when you hit sample size faster.
Low-Traffic Guidance
If your site gets fewer than 10,000 monthly visitors to the page you are testing, standard A/B testing is difficult to do well. Your options:
- Widen your MDE. Only test changes you genuinely believe will produce 20%+ improvements. This means bigger, bolder changes, not button color tweaks.
- Test higher in the funnel. If your checkout page gets 500 visits per month, test your homepage or category page where traffic is higher.
- Use Bayesian sequential testing. Methods like Thompson Sampling or AGILE allow you to update your beliefs as data comes in without the false positive inflation of naive peeking. The tradeoff is that you need more total sample than a fixed-horizon test at equivalent power.
- Run for longer. A 16-week test is not ideal, but it is better than a 2-week underpowered test that gives you a wrong answer you act on.
- Aggregate across segments. If no single page has enough traffic, consider testing a change across an entire content category at once.
There is no statistical trick that lets you get reliable results from a sample that is genuinely too small. The math does not bend. The only real options are: accept higher uncertainty, test bigger changes, or get more traffic.
Key Takeaways
- Calculate sample size before you launch. Everything else is rationalizing.
- Your three inputs: baseline rate, MDE, and power (usually 80%).
- Peeking and stopping early at first significance gets you 40%+ false positives.
- Low-traffic sites need bigger MDEs or longer runtimes, not smaller significance thresholds.
- Sequential testing methods let you monitor in real time but require more total sample.
- Runtime must cover at least one full business cycle, regardless of when you hit sample size.
