🧪 A/B Test Significance Calculator
By Shihab Mia · Updated 2026-08-04
Enter visitor and conversion counts for both variants to run the test.
This A/B test calculator tells you whether the difference between your control (A) and variant (B) conversion rates is real or just random noise. Enter the number of visitors and conversions for each variant, and it runs a two-sample z-test, then reports the conversion rates, the relative uplift, the z-score, the p-value, and a plain "significant" or "not yet" verdict at the standard 95% confidence level. No spreadsheet formulas, no stats course required.
What is the A/B Test Significance Calculator?
An A/B test compares two versions of a page, email, or flow by splitting traffic between them and measuring which one converts better. The problem is that any two random samples will show some difference in conversion rate even if the underlying pages perform identically. A/B test significance testing exists to answer one question: is the gap you are seeing big enough, given your sample size, that it is unlikely to have happened by chance?
This calculator uses a two-proportion z-test, the standard method behind virtually every free A/B testing tool on the market. It pools the conversion rates of A and B into a single combined rate, uses that pooled rate to estimate the standard error of the difference, then divides the observed gap in conversion rates by that standard error to get a z-score. The z-score is converted into a two-tailed p-value using the standard normal distribution, specifically the Abramowitz and Stegun approximation of the error function, which is the same numerical method used in statistical software and spreadsheet ERF functions.
The result is judged against a 95% confidence threshold, meaning a p-value below 0.05 (equivalently a z-score beyond plus or minus 1.96) counts as statistically significant. At that threshold there is less than a 5% chance the observed gap between A and B is a fluke of random sampling. If your result falls short of that bar, the honest answer is not "B wins" or "A wins", it is "not enough evidence yet", and the right move is almost always to keep the test running rather than to call a winner early.
Sample size matters enormously here. The exact same 20% relative uplift, from a 5% conversion rate to a 6% conversion rate, can be statistically significant or completely inconclusive purely depending on how many visitors saw each variant. A small test can show a large, exciting-looking lift that is really just noise, while a well-powered test with the same lift produces a result you can act on with confidence. This is why the raw percentage difference between A and B is never enough on its own, and why an ab test significance calculator like this one, that accounts for sample size, is the right tool for the decision.
When to use it
- Deciding whether a new landing page headline or call-to-action button actually beat the original, or just got lucky.
- Checking an email subject line test before rolling the winning version out to your full list.
- Validating a pricing page or checkout flow change before committing engineering time to a full rollout.
- Reviewing results from a feature flag or gradual rollout to see if the new experience is moving a key metric.
- Comparing two ad creatives or two versions of an app onboarding flow on click-through or activation rate.
- Sanity-checking results a marketing platform reports as a "winner" before trusting the built-in significance claim.
How to use the A/B Test Significance Calculator
- Enter the number of visitors and the number of conversions for variant A, your control or original version.
- Enter the number of visitors and the number of conversions for variant B, the new version you are testing.
- Read the conversion rates, the relative uplift, and the z-score and p-value the calculator computes automatically.
- Check the verdict: if it shows statistically significant, you have a real difference at 95% confidence; if not, keep collecting data before declaring a winner.
Formula & method
Worked examples
A checkout page test runs 5,000 visitors through the control (A) and 5,000 through the new variant (B). A converts 250 visitors (5%), B converts 300 visitors (6%).
- Conversion rates: pA = 250 / 5000 = 0.05 (5.00%), pB = 300 / 5000 = 0.06 (6.00%).
- Pooled rate: p = (250 + 300) / (5000 + 5000) = 550 / 10000 = 0.055.
- Standard error: SE = sqrt(0.055 x 0.945 x (1/5000 + 1/5000)) = sqrt(0.051975 x 0.0004) = sqrt(0.00002079) = 0.004560.
- Z-score: z = (0.06 - 0.05) / 0.004560 = 2.19.
- Two-tailed p-value from the normal distribution: p = 0.0283, which is below 0.05.
Result: z = 2.19, p = 0.0283, a 20% relative uplift that IS statistically significant at 95% confidence.
The exact same 5% versus 6% conversion rates, but from a much smaller test: 1,000 visitors and 50 conversions (5%) for A, 1,000 visitors and 60 conversions (6%) for B.
- Conversion rates: pA = 50 / 1000 = 0.05 (5.00%), pB = 60 / 1000 = 0.06 (6.00%).
- Pooled rate: p = (50 + 60) / (1000 + 1000) = 110 / 2000 = 0.055.
- Standard error: SE = sqrt(0.055 x 0.945 x (1/1000 + 1/1000)) = sqrt(0.051975 x 0.002) = sqrt(0.00010395) = 0.010196.
- Z-score: z = (0.06 - 0.05) / 0.010196 = 0.98.
- Two-tailed p-value from the normal distribution: p = 0.3267, which is well above 0.05.
Result: z = 0.98, p = 0.3267, the same 20% relative uplift is NOT statistically significant, the sample is too small to trust it yet.
Same conversion rates (5% vs 6%), different sample sizes: how significance changes with traffic
| Visitors per variant | A conversions | B conversions | Z-score | P-value | Verdict at 95% |
|---|---|---|---|---|---|
| 1,000 | 50 (5.0%) | 60 (6.0%) | 0.98 | 0.3267 | Not significant |
| 2,000 | 100 (5.0%) | 120 (6.0%) | 1.39 | 0.1654 | Not significant |
| 5,000 | 250 (5.0%) | 300 (6.0%) | 2.19 | 0.0283 | Significant |
| 10,000 | 500 (5.0%) | 600 (6.0%) | 3.10 | 0.0019 | Significant |
Z-score to p-value to confidence level, standard two-tailed thresholds
| Z-score | Two-tailed p-value | Confidence level |
|---|---|---|
| 1.65 | 0.10 | 90% |
| 1.96 | 0.05 | 95% (this calculator's cutoff) |
| 2.33 | 0.02 | 98% |
| 2.58 | 0.01 | 99% |
Common mistakes to avoid
- Stopping the test the moment it looks significant. Checking results daily and stopping as soon as p dips below 0.05 inflates your false positive rate far above 5%, because you are giving randomness many chances to produce a lucky-looking result. Decide on a sample size or a fixed test duration in advance, and only check for significance once you hit it.
- Running an underpowered test and trusting a big-looking lift. A 20% relative uplift from just a few hundred visitors is exciting but usually not significant, as the worked examples above show. Small samples produce noisy conversion rates, so a large percentage gap can still fall well short of the 95% confidence bar. Always check the z-score and p-value, not just the raw percentage difference.
- Treating statistical significance as the same thing as practical significance. A result can be statistically significant, meaning the difference is very unlikely to be chance, while still being too small to matter for the business. A confirmed 0.1 percentage point lift on a low-value page may not be worth the engineering effort to ship. Significance answers "is this real?", not "is this worth doing?".
- Running many tests or metrics and only reporting the ones that hit 0.05. If you test ten metrics or ten variants at once, roughly one will show p below 0.05 purely by chance, even with no real effect anywhere. Pick your primary success metric before the test starts, and treat any other "wins" that show up along the way as hypotheses to re-test, not confirmed results.
Glossary
- Conversion rate
- The share of visitors who complete the goal action, calculated as conversions divided by visitors, for example 250/5000 = 5%.
- Z-score
- How many standard errors apart the two conversion rates are. Larger absolute z-scores mean a bigger, more reliable gap between A and B.
- P-value
- The probability of seeing a difference this large (or larger) between A and B if there were actually no real difference. Lower is stronger evidence of a real effect.
- Statistical significance
- A result where the p-value falls below the chosen threshold, usually 0.05, meaning the observed difference is unlikely to be due to random chance alone.
- Pooled conversion rate
- The combined conversion rate of A and B treated as one group, used to estimate the standard error under the assumption that both variants truly perform the same.
- Relative uplift
- The percentage change in conversion rate from A to B, calculated as (pB - pA) / pA x 100%. A move from 5% to 6% is a 20% relative uplift.
Frequently asked questions
What sample size do I need for an A/B test to be significant?
There is no single number, it depends on your baseline conversion rate and the size of the effect you want to detect. Small lifts on low-converting pages need thousands of visitors per variant, while large lifts on high-converting pages can reach significance with a few hundred. Run this calculator with your actual baseline and expected lift, or use a dedicated sample size calculator before launching the test.
What does a p-value of 0.05 mean in A/B testing?
A p-value of 0.05 means there is a 5% chance of seeing a difference this large between A and B purely by random sampling, even if the two variants actually perform identically. That is the standard cutoff for calling a result statistically significant, giving 95% confidence the difference is real.
Can I stop a test early if it becomes significant?
Not reliably. Checking results repeatedly and stopping the moment p drops below 0.05 is one of the most common A/B testing mistakes, because it lets random noise "win" far more often than 5% of the time. Set a target sample size or test duration up front and stick to it, or use a sequential testing method built for early stopping.
What is the difference between statistical significance and practical significance?
Statistical significance tells you the difference is unlikely to be chance. Practical significance tells you whether that difference is large enough to matter for your business, given the cost of shipping it. A tiny but statistically significant lift on a low-traffic page may not be worth acting on.
Should I use a one-tailed or two-tailed test?
This calculator uses a two-tailed test, which checks whether B is different from A in either direction, better or worse. That is the safer, more conservative default for most A/B tests, since you generally want to know if a change hurt conversions just as much as whether it helped.
What is a good confidence level for A/B testing, 90% or 95%?
95% confidence (a p-value below 0.05) is the standard used by most marketing and product teams, and is what this calculator uses. 90% confidence is more lenient and ships winners faster but accepts a higher false positive rate, so it is best reserved for low-risk, easily reversible changes.
Why does the same percentage lift show as significant in one test but not another?
Because significance depends on sample size, not just the size of the lift. A 20% relative uplift from a small sample of 1,000 visitors per variant can easily be random noise, while the identical 20% uplift from 5,000 or 10,000 visitors per variant is much stronger evidence of a real effect, as shown in the reference table above.