A/B test significance calculator
Find out whether the difference between two versions is real or just chance, and how many visitors you need before you start a test.
What is the A/B test calculator for?
You changed a button colour, a headline or how a price is presented, and the new version looks slightly better. But is the difference a real improvement or just random fluctuation? The human mind is prone to seeing patterns even in chance. This tool helps you base your decision on statistics rather than intuition.
Common mistakes
- Peeking and stopping early: if a test is checked daily and stopped the first time it looks significant, the false-positive rate climbs far above 5 percent.
- Small samples: tests with a few dozen conversions aren't reliable even when they show large differences.
- Too many metrics: look at ten metrics and one may come out “significant” by chance. Choose your primary metric in advance.
- Day-of-week effects: user behaviour varies by day. Run the test for at least one, preferably two, full weeks.
How to use it
- Before starting the test, find the number of visitors you need in the “Sample size” tab.
- When the test reaches that number, enter each version's visitors and conversions in the “Evaluate result” tab.
- Make your decision based on the p-value, the confidence interval and the warnings.
Frequently asked questions
What does the p-value mean?
It is the probability of seeing a difference at least this large by chance alone if there were really no difference between the two versions. For example, p = 0.02 says there is a 2 percent chance of such a difference arising by chance. The p-value is not the probability that B is better.
Can I stop the test as soon as it's significant?
No. Checking results continuously and stopping the first time it looks significant (peeking) seriously inflates the false-positive rate. Fix the sample size in advance and keep the test running until you reach it. Run for at least one full week to capture day-of-week swings.
What is the SRM warning?
Sample ratio mismatch means that while you planned to split traffic 50-50, the actual split is more different than chance can explain. It usually points to a redirect, tracking or bot problem and makes the result unreliable.
Which statistical tests are used?
Significance uses a two-proportion z-test with pooled standard error, the difference uses a Wald confidence interval, each version's rate uses a Wilson interval, and SRM uses a chi-square goodness-of-fit test. The sample size calculation is based on the power formula for the two-proportion z-test.