The Evolution of Scientific Multivariable Testing

Marketers all know testing is important.  Even the most advanced analytics cannot replace the key benefit of testing: proving the real-world impact of changes to the marketing mix.  As Bob Stone put it so well, "Testing is still the best way to find true breakthroughs."

Back in the 1920s, split-run testing in newspapers offered marketers a powerful technique for “scientific advertising” (as Claude Hopkins called it).  Around the same time, a small group of Ph.D. statisticians began using complex mathematics and statistical principles to develop new ways to test many variables at once.  This work began in agriculture, then spread to military, research, and manufacturing operations, and now includes a wealth of research in academic tomes and technical journals.  Only in this last decade have these scientific techniques begun to filter up to the front line of marketing programs.

The core concept is simple: with the right techniques, you change many variables at once and still separate out the impact of each.  This means you can now test dozens of variables, all at once, with greater accuracy, deeper insights, yet using the same small sample size as an A/B split.
 

Good sources for more information

To learn more about the techniques and see examples from other leading marketers, you can look over:

1. An early article on multivariable testing from Direct Marketing magazine (1997), with a good introduction to the techniques and a case study of a catalog test.

2. The recent CM article, The Rising Tide in Circulation Testing, with a brief overview of key benefits and reference to a direct mail case study detailing two tests from a magazine subscriptions direct mail campaign (also published in the 8th edition of Successful Direct Marketing Methods by Stone and Jacobs).

3. Conference slides from the DMA07 session, October 16, 2007, on Direct Mail Testing: Innovations and Insights for Challenging Markets, with case studies from Financial Times and THD. 

4. Conference slides from the 2007 ACCM session on circulation testing, with details of a Lillian Vernon catalog contact strategy test.

5. For those in financial services who enjoy the statistical details, an article from the International Journal of Research in Marketing shows two unique test designs from a credit card direct mail campaign (e-mail us to request a hard copy of the article).
 

You've probably been testing for years, so why take the leap into multivariable testing?

Scientific testing is based on complex mathematics and requires more effort than a simple champion-challenger test, so why bother? 

It's as straightforward as your ABCs...
 

A = Accuracy

Testing numerous marketing elements within one scientific test leads to a dramatic increase in accuracy and confidence in results.  Scientific tests often use about one-tenth the sample size (or reduce experimental error by two-thirds) compared to split-run tests.

For example, the direct mail case study (#2 above) quantified 6 significant effects (top chart, below).  If the same elements had been tested using the same total sample size with split-run techniques (bottom chart), experimental error would have been three times larger (shown by the orange box) and only 1 effect would have been statistically significant.  They would have missed 50% of the gain!

Accuracy also leads to more actionable results.  Retesting significant changes  (i.e., ramp-up, back-testing) is often a good plan, but with greater accuracy, you'll have less uncertainty when you roll-out new creatives, offers, and programs.
 

B = Bottom-line impact

The ultimate measure of success is your testing ROI.  By testing more variables all at once with greater accuracy, you pinpoint more new ways to increase response and profitability.  This improvement multiples as you continue testing. 

You can often test in one campaign what would take 6-18 months using A/B splits.  In each campaign, you have only a certain sample size available for testing.  This often means, statistically, that you can only test 2-3 new ideas if you want statistically-valid results.  Multivariable tests provide equal confidence in each element whether you're testing two or two-dozen variables.  That means that, given the same sample size, a multivariable test offers you greater confidence in results, with no statistical constraints on the number of variables in the test. 

You can test many more ideas.  This often leads to new insights from those "second string" ideas most marketers don't bother testing.  With new insights, you can build a solid advantage your competitors can't match.

Generally, about 1/4th of your new ideas may actually increase response.  If you test 4 ideas, you might find one positive effect.  If you test 12, chances are that you'll uncover 3 positive changes.  Let's see how a series of 12-element multivariable tests would compare to a series of A/B/C/D splits (a fairly conservative comparison).  If each significant effect averages a 5% lift, and multivariable are run only every other month, a comparison of annual improvement is shown below.

This is a simple, yet realistic example of your potential ROI on multivariable testing.  Imagine concentrating a year's worth of learning into one campaign.  With results implemented more quickly, sales and profits continue to grow at an ever-increasing rate.  One scientific test may cost more than one A/B split, but after a few months of testing, your ROI will easily eclipse the incremental cost and effort.

C = Clarity of insights

Scientific tests not only let you test more variables, but also give you deeper insights into complex relationships among marketing-mix elements. 

Results quantify:
     ●  Main effects: the impact of each change on its own
     ●  Comparative effects: which elements have more/less impact
     ●  Interactions: how main effects change depending upon how other elements are set

A 4-element price/offer test from the case study, above, quantified three significant main effects and one two-way interaction, shown below.

Main effects showed that the low subscription price (A+) and 1-year subscription term (B-) increased response (by 23% and 12% respectively).  However, this two-way interaction leads to a new opportunity.  If the subscription price is kept low (two points at right), then the response rate is about the same for a 1-year or 18-month subscription.  The price-term 2-way interaction offers the opportunity to increase the subscription term (and revenue) without hurting response.  This would be impossible to see with split-run tests.
 

S = Speed

In one campaign—with the same sample size—you can test two or two-dozen marketing-mix elements with equal confidence. 

Sample size is normally a serious constraint in catalog and direct mail tests.  With split-run techniques, marketers often have only enough volume to test 2-3 new covers, packages, creatives, or price points in each campaign.  Scientific testing works differently.  As long as all variables are part of the same test design, you can use the same small sample size no matter how many elements you're testing.

As one B2B marketer said, "We learned in 3 months what would take 8 years with A/B splits."  With small infrequent mailings, the marketing team would have needed about 15 campaigns to test the same ideas, with equal confidence, as they tested in one campaign with scientific techniques.
 

If you agree that "testing is still the best way to find true breakthroughs"...

If testing is an important part of your catalog, direct mail, and other marketing programs, it may be a good time to leverage the most efficient, powerful, and insightful techniques you can find.  See your marketplace more clearly with scientific testing.

If you would like more information, feel free to contact us with any questions and to discuss your unique marketing challenges.

Back to top