A/B testing is a powerful way to improve your online marketing.
But more important than any improvement you might get from testing is what you learn from your testing that you can apply in the future. That means the process behind your test is critical.
You should take these three basic steps before running any test:
1. Choose what element of your program you want to improve
For example, you might want to strengthen your e-mail newsletter or your donation page.
2. Identify how you measure success
You don’t run a test just to see what happens. You’re trying to improve something. What is that something?
So if you’re optimizing your newsletter, what’s the goal of the newsletter? To drive someone to your website, perhaps? Or what’s the goal of your donation form? To capture the most gifts or to capture the most revenue?
You will then measure your test results based on this goal.
3. Develop a hypothesis about how you will improve that measure
This step is critical. With a clear hypothesis—“more links in my newsletter will drive more traffic to the site,” for example, or “less clutter on the donation page will lead to more gifts”—you have a testable proposition. Your test will either confirm or reject your hypothesis, and you can apply that lesson in the future.
The debate between fully-open and fully-enclosed work spaces misses the point, David Craig argues. What really matters is flexibility:
The bigger flaw, though, in recent criticisms of open workplaces is the underlying idea that there’s only one choice: open or enclosed. Work is invariably a combination of individual work, collaboration, coördination, creativity, and other things, all of which can take a variety of forms, sometimes in just one person in one day. As research done by CannonDesign with 14 organizations over the past year has shown, the average employee does want fewer distractions, but they also want 35% more frequent interactions within their teams; they want more energy and buzz in the workplace than less, but they also want the flexibility to escape to a quiet place from time to time. What they definitely don’t want is one space that’s just open or just enclosed.
This is very much in line with what Jason Fried and David Heinemeier write in their book Remote.
Jason Fried explains why he prefers simple, text-based web designs to slick interfaces:
None of which is to say that a text-heavy design is the right solution for everyone. But I’ve always found it interesting that some of the most popular sites on the Web–Amazon, eBay, Craigslist, Wikipedia, to name a few–are often very heavy on the text and very light on the imagery. These sites won’t win any design awards, but they seem to communicate very clearly to their intended audience.
At the end of the day, you’re designing for your customers, not for other designers.
The New York Times:
Social media and Big Data, the term du jour for the collection of vast troves of information that can instantaneously be synthesized, are supposed to help us make smarter, faster decisions. It seems as if just about every C.E.O. of a global company these days is talking about how Big Data is going to transform their business. But with increasing frequency, it may be leading to flawed, panic-induced conclusions, often by ascribing too much value to a certain data point or by rushing to make a decision because the feedback is available so quickly. This digital river of information is turning normally level-headed decision-makers into hypersensitive, reactive neurotics.
One big danger is in using the canned reports that come with e-mail providers or services like Facebook. These reports may not align with your actual goals and can lead you astray. E-mail programs, for example, usually measure the success of a message by open rate—even though open rate usually isn’t the goal of your message. Facebook’s reports on the likes and shares generated by a link you posted also won’t be of much help if your goal is driving traffic to that link.
Data should inform your decisions and allow you to confirm or disconfirm hypotheses. It shouldn’t make your decisions for you.
Have you ever been led astray by incomplete or poorly-represented data?
Are you basing your online marketing plans on the latest benchmark study? What on earth for?
A.G. Lafley and Roger Martin explain why this is foolish in Playing to Win:
Every industry has tools and practices that become widespread and generic. Some organizations define strategy as benchmarking against competition and then doing the same set of activities but more effectively. Sameness isn’t strategy. It is a recipe for mediocrity.
Benchmark studies can be interesting sources of inspiration and ideas. But they’re not a how-to manual, and you certainly shouldn’t measure yourself against them. You should be doing what’s best for your audience, not aping what your competitors are doing with theirs.
As Flint McLaughlin puts it, “best practices on the internet are typically pooled ignorance.”
Peep Laja identifies eleven common A/B testing mistakes (his headline says twelve, but the article is missing a ninth item):
- A/B tests are called early
- Tests are not run for full weeks
- A/B split testing is done even when they don’t even have traffic (or conversions)
- Tests are not based on a hypothesis
- Test data is not sent to Google Analytics
- Precious time and traffic are wasted on stupid tests
- They give up after the first test fails
- They don’t understand false positives
- They’re running multiple tests at the same time with overlapping traffic
- They’re ignoring small gains
- They’re not running tests at all times
In my experience, the first, fourth, and tenth mistakes are easiest to make. I’ve made them myself in my impatience to get a result, my desire to just “try something,” or my desire for a big lift.
But cutting corners to get a big lift as quickly as possible doesn’t teach you anything you can use in the future—and learning is the most valuable takeaway from any test.
Read the whole thing.
What mistakes have you made when testing?
Upworthy succeeds in driving piles of traffic not because because it uses some sophisticated new technology.
Upworthy succeeds because it recognizes that its readers are human beings, not abstract “traffic” or “eyeballs.” This is an old technique applied to the web.
As Derek Thompson explains in his report on Upworthy’s success, its writers select stories that have emotional appeal and craft headlines to pique our curiosity. They combine this with optimization techniques to find out what works best:
Upworthy has mastered the dark viral arts with a unique blend of A/B technology and lily-white earnestness. The staff scours the Web for “stuff that matters,” writes multiple headlines for a test audience, selects the top-performer, and blasts it out on social media. It’s a deceptively simple plan that’s devouring the Internet, one Facebook Newsfeed at a time. The site nearly surpassed 50 million unique visitors in October, which suggests traffic comparable to giants like Time.com, and Fox News. …
What’s the “secret”? An entertaining slideshow of Upworthy’s headline-writing strategies last year repeatedly references the “curiosity gap.” The idea is both to share just enough that readers know what they’re clicking and to withhold just enough to compel the click.
That linked slideshow is definitely worth reading through.
Adam Penenberg challenged the white-hat hackers at SpiderLabs to penetrate his digital life. They found fewer ways to break in than at their corporate clients—but once in, they had access to everything:
With me, however, there were fewer paths that could lead to the mother lode: my laptop, email, bank, social media accounts, and home. Once in, though, his team found few firewalls protecting my data, and mostly in the form of a pastiche of passwords and log-in credentials. These, I quickly learned, were not secure.
Are you looking to raise money online? These four tips may be helpful:
- Make sure people can donate on your web site
- Build an effective e-mail program
- Make sure you’re mobile-ready
- Use social media to build relationships, not to drive gifts
But the number one trick to being a great online fundraiser? Assume you have no idea what actually works.
I elaborated on these points in a presentation last week at the Leadership Institute. My full slide deck is below.
The adage “learn from your mistakes” is a bit misleading, Mig Reyes argues. Instead of trying to fail, we should just try:
The makers of our world would be better off mimicking scientists with their work. Harp on deliberate practice. Reinvent their processes daily. Share every discovery. And most importantly, try new things often…
When everything’s an experiment, you shed the fear that comes with trying new things.
Consistent experimentation is the key to success, whether in marketing or any other field.