Free A/B Test Sample Size Calculator

Running a test without enough visitors is like asking three people if your shirt looks good. Technically feedback. Not exactly reliable.

This calculator tells you the minimum number of visitors you need per version before your A/B test results actually mean something. Plug in your current conversion rate, the smallest improvement you’d care about, and how confident you want to be. You’ll get a number you can plan around.

How to use this calculator

1. Enter your current conversion rate. Check Google Analytics if you’re not sure. Most sites sit between 1% and 5%.
2. Set the smallest improvement worth detecting. If you’d only change your page for a 20% boost, enter 20%.
3. Pick your confidence level. 95% is standard. Go higher if the stakes are high.
4. Read the result. That’s how many visitors you need per version (not total).
5. Divide by your daily traffic to estimate how long the test needs to run.

How we calculate this

The math behind this calculator is called a two-proportion z-test. Here’s what that means in plain English.

You’re comparing two groups: visitors who see your current page (Version A) and visitors who see the changed page (Version B). The question is: “If Version B really is better, how many people do I need to watch before I can tell the difference from random luck?”

Three things determine that number:

Your current conversion rate sets the starting point. A site converting at 2% needs more visitors than one converting at 10%, because small differences are harder to spot when the numbers are already small. Think of it like weighing something on a kitchen scale versus a bathroom scale. The kitchen scale catches smaller changes.

The smallest improvement you’d care about (experts call this the minimum detectable effect). If you only care about catching a 20% relative improvement, you need fewer visitors than if you’re trying to catch a 5% improvement. Bigger differences are easier to spot.

Your confidence level is how sure you want to be. At 95% confidence, there’s a 5% chance you’ll see a “winner” that isn’t actually better. At 99%, that drops to 1%, but you’ll need more visitors to get there.

The formula combines these inputs using the normal distribution (the bell curve you’ve seen in textbooks). It calculates the overlap between two bell curves, one for each version, and figures out how many data points you need before the curves stop overlapping enough to tell them apart.

Here’s the actual formula, simplified:

n = (Z_α/2 + Z_β)² × (p₁(1-p₁) + p₂(1-p₂)) / (p₁ - p₂)²

Where n is visitors per version, Z values come from your confidence and power settings, p₁ is your current conversion rate, and p₂ is the rate you’re trying to detect. You don’t need to memorize this. That’s what the calculator is for.

The key insight: sample size grows exponentially as you try to detect smaller differences. Catching a 20% improvement needs about 4x fewer visitors than catching a 10% improvement. This is why we recommend testing big, obvious changes (headlines, page layouts) over small tweaks (button colors). Bigger changes produce bigger differences, which means faster, cheaper tests.

FAQ

How long should I run an A/B test?

Divide the sample size by your daily visitors per version. If you need 12,000 visitors per version and you get 500 per day, that’s 24 days. Always run for at least one full week to account for weekday and weekend differences, even if you hit your sample size sooner. Two to four weeks is typical for most sites.

What sample size do I need for an A/B test?

It depends on three things: your current conversion rate, the size of the improvement you want to detect, and how confident you want to be. A site with a 3% conversion rate trying to detect a 20% relative improvement at 95% confidence needs roughly 12,000 to 15,000 visitors per version. Use the calculator above to get your specific number.

What happens if I stop a test early?

You’ll get unreliable results. Early in a test, random fluctuations look like real patterns. A test might show Version B “winning” by 30% after 500 visitors, then settle to a 2% difference after 5,000. This is called peeking, and it’s one of the most common A/B testing mistakes. Decide your sample size before you start and stick to it.

Does sample size change if I have low traffic?

The required sample size stays the same. What changes is how long it takes to get there. If you only get 200 visitors a day, a test that needs 12,000 visitors per version will take 60 days. At that point, consider testing bigger changes so you can use a larger minimum detectable effect, which shrinks the sample size. Or focus on your highest-traffic pages first.

Should I use one-tailed or two-tailed testing?

Two-tailed is the safer default. It checks whether Version B is better or worse than Version A. One-tailed only checks one direction, so it needs fewer visitors, but you’ll miss it if your change actually hurt conversions. Most practitioners and statistical best practices recommend two-tailed unless you have a strong reason not to.

What’s a good minimum detectable effect to use?

Start with 20% relative improvement. That’s realistic for meaningful changes like new headlines or restructured pages. If you set it too low (like 5%), you’ll need a massive sample size. If you set it too high (like 50%), you might miss real improvements that are worth capturing. The right MDE depends on your traffic and patience. The MDE guide breaks this down further.

Ready to run your test? Once you know your sample size, set it up in Kirro. Takes about 3 minutes. No code, no developer needed.