Are you using A/​B testing in your fundraising campaigns?

Brian Christian explains how firms are using A/​B testing to improve their online marketing:

Over the past decade, the power of A/​B testing has become an open secret of high-stakes web development. It’s now the standard (but seldom advertised) means through which Silicon Valley improves its online products. Using A/​B, new ideas can be essentially focus-group tested in real time: Without being told, a fraction of users are diverted to a slightly different version of a given web page and their behavior compared against the mass of users on the standard site. If the new version proves superior—gaining more clicks, longer visits, more purchases—it will displace the original; if the new version is inferior, it’s quietly phased out without most users ever seeing it. A/​B allows seemingly subjective questions of design—color, layout, image selection, text—to become incontrovertible matters of data-driven social science.

Google runs thousands of such tests every year to improve its search results and other products. In fact, Google runs so many simultaneous tests, Christian says, that “the percentage of users getting some kind of tweak may well approach 100 percent.” Using these same principles, the Obama campaign in 2008 used A/​B testing to increase e-mail signups by 40 percent.

It ran similar tests in the 2012 cycle.

Christian points out, though, that A/​B testing is best for identifying incremental changes rather than the overhauls that may be necessary. In other words, effective marketing remains an art as well as a science. Marketers and web designers “may find themselves chasing ‘local maxima’—places where the A/​B tests might create the best possible outcome within narrow constraints—instead of pursuing real breakthroughs.”

Tell me in the comments: how have you used A/​B testing?