Split-run techniques—often called A/B splits, test-control, champion-challenger, or one-variable-at-a-time testing—have given marketers some valuable answers, but split-run techniques are like a Model T to today’s Ferrari…

The science of testing has evolved far beyond split-run techniques

With the right techniques, you can rapidly test dozens of marketing-mix variables, all at once, with about the same sample size as an A/B split, yet with much greater accuracy, plus in-depth results that quantify every variable on its own and in combination with others. Here’s just one example of performance before, during, and after one 18–element test:

“Almost any question can be answered, cheaply, quickly and finally, by a test campaign.”
– Claude Hopkins, Scientific Advertising (1923)

Since Claude Hopkins wrote Scientific Advertising in 1923, marketers have championed the split-run approach… and overlooked significant innovations. Like an advanced civilization hidden in the jungle, the specialized field of “scientific testing” has flourished for years out of sight of most of the marketing world. Unfortunately, experts developing these new techniques largely ignored the marketing world, while complex statistics—in 700+ page tomes covering subjects like nongeometric designs, Galois fields, confounding schemes, reflection, and effect heredity—made this vast realm of scientific testing seem out of reach.

Fortunately, newly-refined statistics and market-focused strategies now offer an abundance of testing alternatives with little additional effort. Split-run tests are only the simplest of myriad options that offer the freedom to test more variables, more rapidly, and with greater accuracy. Advantages include:

Back to top

1. Quantity – You can test up to three-dozen variables simultaneously in one test (versus each variable as a separate test with split-run techniques). By casting a wide net and testing more marketing-mix elements, you can pinpoint numerous changes proven to increase sales. Often, elements may have little effect on their own, but 5 or 6 together give you a significant jump in sales. If you believe the marketing mantra, “the more you test, the more you learn,” then scientific testing finally gives you the tools to test 10 or 20 variables at once, while your competitors are stuck testing 1 or 2.

One e-mail test of 12 variables across 3 different customer segments used only 34,000 total names (for all test cells and the control). Implementing the optimal combination, sales jumped 41%.

2. Speed and sample size – You can use the same small sample size whether you are testing 2 or 22 marketing-mix elements. More efficient statistical techniques let you learn more from every piece of data.

Testing 22 elements of a banner ad—including messages, offer, graphics, layout, colors, buttons, and shape—one 3-week test pinpointed 8 changes for a 72% increase in sales. Split-run techniques would have required more than 60 weeks of testing.

3. Depth of insights – Results quantify the real-world impact of every test element on its own, plus interactions, where elements may have a different impact in combination with others.

Quantifying each variable on its own allows you to rapidly optimize marketing programs—implement changes that help, avoid changes that hurt, and use the lower-cost option for elements that make no difference.

In addition, interactions give insights impossible to uncover with split-run testing. For example, price-offer interactions often lead to impressive gains in profitability:

• One interaction showed that a 15% discount and free gift each increased response, but both together were no better than the discount alone.

• Another test quantified interactions among price points, shipping & handling, and special offers, helping the marketing team increase profitability beyond what years of split-run tests had shown.

4. Flexibility – A wide variety of test designs allow you to create the most powerful and insightful test to meet your objectives, whether testing many combinations of a few key elements (like price points), or testing dozens of creative changes in a circular, catalog, or postcard campaign.

Test design terminology may seem like a foreign language, but some good choices include:

• Full-factorial design for price-offer testing, where interactions are valuable

• Plackett-Burman design for direct mail, minimizing the number of test cells

• Central composite design to test multiple levels and curvature

• Reflected/resolution IV design to increase accuracy for retail testing

• Split-run test to add a 3rd level to one element in a fractional-factorial design (or to test a larger catalog or completely new creative concept)
   

The immense power and efficiency of scientific testing has given a few market leaders a formidable competitive advantage, but it does not, on its own, guarantee success. Like marketing itself, testing looks so easy when done well. But if you want to hit the ground running, it’s important to get some guidance.

If you want to understand how these tests work, look over How to test many variables at once, or read over some of the Case studies & articles explaining what other industry leaders have achieved.

Back to top