In short
- Plan before you start. At a 1.4% conversion rate and 5,000 visitors a week, finding a 25% lift takes about 8 weeks: 19,862 visitors per version.
- A result counts when it is significant - a p-value below 0.05, or 95% confidence - and the test ran its planned visitors and full weeks.
- Small stores should test big changes. Doubling the lift you look for cuts the visitors you need to about a quarter.
How to use the A/B test calculator
- Plan a test: enter the visitors the tested pages get in a week, your conversion rate and the smallest lift worth finding. You see the visitors per version and the weeks.
- Run the test until it reaches those visitors - in full weeks, without stopping early.
- Check my result: enter the visitors and orders of each version. You see whether B won, lost, or is not proven yet. Copy the link to keep the result.
How long should an A/B test run?
Weeks a test needs at the average Shopify conversion rate of 1.4%, by the visitors the tested pages get and the smallest lift you want to find:
| Visitors a week | 10% lift | 20% lift | 30% lift | 50% lift |
|---|---|---|---|---|
| 1,000 | 2+ years | 61 weeks | 29 weeks | 12 weeks |
| 2,500 | 93 weeks | 25 weeks | 12 weeks | 5 weeks |
| 5,000 | 47 weeks | 13 weeks | 6 weeks | 3 weeks |
| 10,000 | 24 weeks | 7 weeks | 3 weeks | 2 weeks |
| 25,000 | 10 weeks | 3 weeks | 2 weeks | 2 weeks |
Every test runs at least two full weeks, so each weekday is in both versions. Anything past eight weeks is too long to be practical: seasons, ads and prices change while it runs.
How the sample size is calculated
p₁ is your rate now, p₂ the rate after the lift. 1.96 and 0.84 set the usual 95% confidence and 80% power. At 1.4% and a 25% lift (p₂ = 1.75%): 19,862 visitors per version.
Power is the chance the test finds a lift that is really there. At 80%, one real winner in five still looks like "no difference" - one more reason to test changes big enough to matter.
How to read the result
The calculator runs a standard test for two conversion rates. Three numbers matter:
- Lift: how much better or worse B converts than A. 1.40% → 1.80% is a lift of 28.6%.
- p-value: the chance of a difference at least this big if the two versions really sold the same. Below 0.05, the result is significant.
- Confidence: 1 − p-value, as a percentage. 95% or more is the usual line.
A p-value is not the chance that B is better, and 95% confidence does not promise the same lift after launch - the lift you measured is an estimate, and the real one is often smaller.
Rules for a fair A/B test
- One change at a time, so you know what made the difference.
- Decide the metric, the visitors and the weeks before you start - orders or sales per visitor, not clicks.
- Run full weeks, at least two, and don't stop early when one version leads.
- Avoid sale weeks and holidays, when shoppers behave differently.
- Expect most ideas to fail. In Optimizely's analysis of 127,000 experiments, only 12% produced a significant win on their main metric.
What to test first on a Shopify store
Test changes big enough to move orders, at the step that loses the most buyers. Find that step with the leak finder in the conversion rate calculator. Good first tests:
- Shipping costs and delivery dates on the product page instead of at checkout.
- A free shipping threshold a little above your average order.
- New product photos: on a model, in use, at real size.
- Reviews near the buy button.
- A shorter product page on phones, with the buy button on the first screen.
The conversion rate guide lists the ten most common leaks and how to fix each one. On Shopify, Rollouts (Grow plan and above) runs a test between two versions of your theme with no app.
Not enough traffic for an A/B test?
Make the change and compare four full weeks before with four full weeks after, in the same season, with no other change in between. It is weaker evidence than an A/B test, but much better than guessing. Keep a change log - one line per change, with the date - so you know what to compare.
What this calculator does not tell you
- Whether the test was set up right. Uneven traffic between versions, a broken page in one of them, or bots in one group give a wrong result with a convincing p-value.
- Order value. The test compares conversion rates. A version that sells more orders of smaller value may earn less - check sales per visitor too.
- The long run. A discount banner can win a two-week test and train shoppers to wait for discounts.
By Bart Ziem, Adeagle · Method: two-sided test for two proportions, 95% confidence, 80% power - the defaults of common testing tools · Shopify benchmark: Littledata, via Nudgify · Experiment win rate: Optimizely, 127,000 experiments · Examples, not promises