MSM FULL-FUNNEL CONVERSION GLOSSARY
What is A/B Testing?
A/B testing is a method of comparing two versions of a page, email or advert by showing each to a different group at the same time and measuring which produces more of the outcome you want.
THE SHORT ANSWER
The Definition
A/B testing is a method of comparing two versions of a page, email or advert by showing each to a different group at the same time and measuring which produces more of the outcome you want.
Why It Matters Commercially
Done properly it replaces opinion with evidence and stops expensive redesigns based on whoever argued hardest. Done badly it produces confident conclusions from data that can't support them, which is worse than not testing at all.
How It's Measured
There isn't a single formula. What matters is statistical significance, usually set at 95% confidence, and sample size. Both are calculated by testing tools, and a result reached before either threshold is met isn't a result.
Who Owns It
Marketing usually runs the tests. Whoever owns the website needs to sign off what gets tested, because a test that changes a page mid campaign muddies both the test and the campaign.
HOW IT WORKS
A/B testing shows two versions of something to different groups of visitors at the same time and measures which performs better against a defined goal.
Running one properly means four things:
- A hypothesis. Not "let's try a different headline" but "visitors aren't enquiring because the headline doesn't say who this is for, so a clearer headline will increase enquiries."
- One variable. Change the headline or the button, not both, or you won't know which caused the difference.
- Enough traffic. The test needs sufficient visitors and conversions to reach statistical significance. Testing tools calculate this and it's usually higher than people expect.
- A full cycle. Run for complete weeks. Behaviour differs between weekdays and weekends, so stopping mid week skews the result.
Why most small business A/B tests fail
Not because the idea is wrong, but because of arithmetic. Statistical significance depends on the number of conversions, not visits. A site with 500 monthly visitors converting at 2% produces around 10 conversions a month. Split across two variants that's 5 each, which is nowhere near enough to distinguish a real effect from chance.
Businesses at that scale stop tests early, see a difference, and act on noise. The honest position is that A/B testing needs volume, and below a certain level your effort is better spent on diagnosis than experimentation.
A/B testing versus multivariate testing
A/B testing compares two complete versions. Multivariate testing changes several elements at once and works out which combination performs best.
Multivariate gives you more information per test and needs far more traffic to do it, because every combination requires its own sample. For most B2B businesses it isn't realistic. A/B testing is demanding enough.
A/B testing versus conversion rate optimisation
CRO is the discipline. A/B testing is one method within it, and not the first one to reach for. Diagnosis comes first: analytics, session recordings, form drop off, speaking to customers. That work tells you what's worth testing. Testing without it is guessing with extra steps.
Our CRO work starts with diagnosis for that reason. There's more in why CRO is bigger than your website.
WHAT THIS LOOKS LIKE IN PRACTICE
A business wants to test two homepage headlines. They get 1,200 visitors a month to that page and it converts at 1.5%, so 18 enquiries.
Split across two variants, each version gets 600 visitors and around 9 conversions. To detect anything short of an enormous difference at 95% confidence, they'd need to run the test for several months.
Two weeks in, version B is ahead. It looks decisive and it isn't. At those numbers the gap is well within the range you'd expect from chance alone. Acting on it means rebuilding the site around a coin toss.
IS THIS COSTING YOU REVENUE RIGHT NOW?
Before you run another test:
- Do you know how many conversions you need for a reliable result?
- Does each test have a written hypothesis, or is it a list of things to try?
- Are you changing one variable at a time?
- Do tests run for complete weeks?
- Do you record the results of tests that showed no difference?
- Did diagnosis suggest this test, or did somebody suggest it in a meeting?
Two or more no answers and your tests aren't producing findings you can rely on.
SEE WHAT YOUR COMPETITORS ARE DOING THAT YOU AREN'T
We'll analyse two of your competitors against your own site and show you where they're winning conversions you should be getting.
WHAT WE SEE IN THE FIELD
A/B testing gets recommended to businesses that don't have the traffic to use it, and they spend months running tests that can't produce a reliable answer.
If you get under a few thousand relevant visitors a month, spend the effort on diagnosis instead. Watch session recordings. Look at where forms get abandoned. Ask three customers what nearly stopped them. That will tell you more than a test you have to stop early.
Reviewed by Ian Wilson, MSM.
Frequently Asked
KEEP EXPLORING
FIND OUT WHERE YOUR MARKETING AND SALES ARE LOSING CONVERSIONS
Book your 45 minute dicovery call where we will discuss and examine your customer journey from attention and website behaviour through lead quality, handover, CRM, pipeline, follow up and close, then tells you what to fix first.