What you should remember when running A/B tests
A/B testing can help you make great leaps in optimizing your online marketing, but it’s not a panacea. It requires a lot of measurement, commitment and patience.
Peep Laja offers three useful warnings:
- Most A/B tests won’t produce huge gains (and that’s okay)
- There’s a lot of waiting (until statistical confidence)
- Trickery doesn’t provide serious lifts, understanding the user does
This is spot on. Most of the tests I have run have failed to achieve any lift, while others were inconclusive statistically even after collecting considerable data. And his final point is critical: focus your testing not on the quick win but on how you can best convince your customers to buy.
If I had to add a fourth point, it’d be this: Run tests only to learn something, not simply for the sake of testing. What question are you trying to answer with your test, and how would the results lead you to do things differently in the future?