‘Realistically, do you care?’
“For most websites, ‘change only one thing per test’ is pretty bad advice,” Alex Birkett writes at Conversion XL:
While you might get precise knowledge about what specific element caused a lift when you test one element at a time, you may be missing the forest for the trees.
“Let’s say you change 100 things, and sales go up 30% – but you don’t know what caused the change,” Peep Laja says in the article. “Realistically, do you care?”
You’re probably leaving money on the table by not running radical tests. One-element tests can limit the scale of your lifts—they can hit local maxima, in optimization speak—and can take far longer to yield meaningful results.