A/B Testing & Conversion Optimization

What is A/B testing?

A/B testing compares a control version with a variant by randomly splitting traffic, to see whether a change causes a better result. It is a core method of conversion optimization. A valid test fixes the metric, sample size and duration in advance and avoids stopping early, so the result reflects the change rather than chance.

By Ankush Agarwal · Last updated October 9, 2026

Who it is for

Is this the right fit?

Teams that run tests but do not trust the results

Stores with limited traffic that want to know what they can learn

Organizations building an experimentation program and roadmap

The problem

What it looks like when this is not working

Tests are stopped as soon as one variant looks ahead

Early results swing widely, so stopping at a good moment makes false winners far more likely.

Nobody checks sample size, so small lifts are declared winners

Without enough visitors, differences are mostly noise, and decisions rest on chance.

Results are not logged, so the same ideas are retested

Learning is lost when people leave, and the team repeats work it has already done.

Test ideas come from opinion rather than evidence

The loudest idea wins instead of the one most likely to matter.

My approach

How I work, step by step

Hypothesize

Each idea starts with evidence from analytics, research and customer feedback, and is written as a hypothesis: what we will change, for whom, and what result we expect and why.
Output: Prioritized, evidence-based hypotheses
Design

I choose the primary metric and guardrails, and size the test before launch: baseline rate, the smallest lift worth detecting and how long it must run.
Output: Test design with a sample-size and duration plan
Run

One change at a time, a random split, full weekly cycles, a check of the traffic split and no peeking. These habits protect the validity of the result.
Output: A valid, well-monitored test
Analyze

Results are read with the planned method. I look at practical as well as statistical significance, and at the guardrail metrics.
Output: A readout with a clear decision
Learn

Every result, including no-difference results, goes into a test log that feeds the next round of ideas.
Output: Test log and updated roadmap

What you get

Deliverables

Experimentation roadmap and prioritization

A ranked backlog of tests, with the evidence behind each.

Test designs with sample-size and duration plans

What to test, how many visitors it needs and how long it will take.

Analysis and readout templates

Consistent, readable reporting for stakeholders.

Test log

A searchable record of what was tested and what was learned.

Playbook with guardrails and a QA checklist

The rules that keep tests valid.

Team training on test design

Practical sessions on hypotheses, sample size and reading results.

Outcomes

What changes for your team

Trustworthy results

Decisions rest on valid evidence

Fewer false winners

Planned sample sizes and no peeking protect against chance

Faster learning

A prioritized roadmap and a test log compound insight

A shared playbook

The team runs tests the same way every time

Tools and concepts

A/B testing platforms Power analysis Sample ratio mismatch checks Guardrail metrics Conversion optimization Test logs

Common mistakes

What to avoid, and what to do instead

Peeking and stopping early

Fix sample size and duration in advance and read the result once.

Testing tiny changes with little traffic

Test bigger changes and use higher-volume tests such as ads and emails.

Running overlapping tests on one page

Keep tests separate so their effects are not mixed.

Celebrating wins and ignoring losses

Log every result; no difference is still learning.

Ways to work together

Choose the level of help you need

Program assessment

A review of how you test today, with a prioritized improvement plan.

Roadmap and test design

A prioritized backlog and rigorous test designs for your top ideas.

Experimentation partnership

Ongoing test analysis, readouts and playbook coaching.

Key terms

Plain-language definitions

Control and variant: The original experience and the changed experience a test compares.
Minimum detectable effect: The smallest improvement a test is designed to reliably detect.
Statistical power: The chance a test detects a real effect of the planned size; 80% is a common target.
Guardrail metric: A metric watched to make sure a winning variant is not harming something else, such as AOV or refunds.

FAQ

Frequently asked questions

How long should an A/B test run?

Long enough to reach the planned sample size and to cover full weekly cycles, so day-of-week effects balance out. Many tests run at least two full weeks. Decide the duration in advance rather than stopping when the result looks good.

How much traffic do I need for an A/B test?

It depends on your baseline conversion rate and the smallest lift you want to detect. For example, a 2% baseline needs roughly 21,000 visitors per variant to detect a 20% relative lift at common settings (95% confidence, 80% power). Smaller lifts need far more.

What is a sample ratio mismatch?

A sample ratio mismatch is when the actual split of visitors between variants differs noticeably from the planned split, such as a 50/50 test delivering 55/45. It usually signals a setup or tracking problem, so results from that test should not be trusted until it is explained.

What is statistical significance?

Statistical significance indicates that an observed difference is unlikely to be due to chance alone, judged against a threshold chosen in advance, commonly 5%. It does not say how large or valuable the difference is, so practical impact should be considered too.

What can we test with low traffic?

Test bigger changes, such as the offer or value proposition, and use higher-volume tests like ad creative or email subject lines. Combine them with qualitative evidence such as session recordings, user tests and surveys, and treat before-and-after comparisons as hints rather than proof.

Why shouldn't I stop a test early?

Early results swing widely, and checking repeatedly while ready to stop on a good result makes false winners far more likely. Fixing the sample size and duration in advance keeps the error rate at the level you planned for.

In practice

Good experimentation is mostly discipline: a hypothesis grounded in evidence, a test sized before launch, and a result that is read the way it was planned. The tools matter less than the habits, and the most valuable asset an experimentation program builds is a log of what has and has not worked.

Low traffic is not a reason to skip testing, but it changes what you test and how you decide. Larger changes, higher-volume tests such as ad creative and email, and qualitative evidence all help a smaller team learn faster without fooling itself.

Working on Experimentation and Optimization?

I am glad to compare notes. The best way to reach me is on LinkedIn.

Connect on LinkedIn →

Experimentation and Optimization