A/B testing is the best way to replace opinion with evidence. It also demands more traffic than most new stores have, which is why so many small-store tests produce confident but wrong answers.
The math behind a valid test
The visitors you need in each variant depend on two things: your baseline conversion rate and the smallest lift you want to be able to detect. The smaller the lift, the more visitors you need, and the number grows quickly.
| Baseline conversion | +10% lift | +20% lift | +30% lift | +50% lift |
|---|---|---|---|---|
| 2% | 80,681 | 21,108 | 9,797 | 3,825 |
| 3% | 53,210 | 13,914 | 6,454 | 2,517 |
| 5% | 31,233 | 8,158 | 3,780 | 1,470 |
Visitors needed per variant, at 95% confidence and 80% power (double it for total traffic). A store converting at 2% needs roughly 21,000 visitors in each variant to reliably detect a 20% relative lift, from 2.0% to 2.4%. A store with 5,000 total visits cannot learn that from a single-page test.
What to do instead
- Test big, not small. Headline and value proposition, offer, hero media, guarantee placement and shipping threshold. Button colors will not move the needle.
- Use higher-volume tests. Ad creative and email subject lines produce far more impressions and sends than store visits, so they reach valid results sooner.
- Lean on qualitative evidence. Session recordings, a handful of short user tests, support tickets, review mining and surveys.
- Treat before-and-after comparisons as hints. Note what else changed, such as traffic source or promotions.
Rules for any test
- Change one thing at a time and split traffic randomly.
- Choose the primary metric before you start, and track guardrails such as AOV and refunds.
- Decide sample size and duration in advance, and run full weeks.
- Do not stop early because one variant looks ahead.
- Check that the traffic split is close to what you intended; a lopsided split means something is broken.
- Log every test, including the ones with no winner. “No difference” is a result.
