Proven success over the last decade

The first multivariable Internet test was an online advertising test from the "dark ages" of the Internet.  This one 3-week test of 22 elements—including messages, offer, graphics, layout, colors, buttons, and shape—pinpointed 8 changes for a 72% increase in conversion rate.  Common split-run techniques would have required more than 60 weeks of testing and completely missed a valuable two-way interaction.

Around that time, articles on scientific Internet testing were first published in Inter@ctive Week and Target Marketing and the use of multivariable techniques grew rapidly in e-mail, online advertising, landing page (and website), and search (SEO and SEM) marketing programs. 

For a quick overview of scientific testing techniques and benefits, you can look over:
     ●  A recent article from Internet Retailer
     ●  A detailed case study of a multivariable Landing Page Test
     ●  Conference slides from the DMA06 session on multivariable testing, with details of an 11-element Conde Nast e-mail test.

Speed, flexibility, and power

Advanced techniques are ideal for Internet testing.  Production costs are low, flexibility is high, and results can be implemented within days or weeks.  This means you can be more aggressive in testing numerous marketing-mix elements quickly, clearly, and cost-effectively.  But simple A/B splits are so much easier, so why do something new?

1. Speed

You can complete a scientific test in a fraction of the time required for statistically-valid A/B splits.

One Proflowers landing page test (presented at the 2003 Direct Marketing Association annual conference) included 23 marketing-mix elements tested over 3 weeks.  Split-run tests would have required more than one year of testing to achieve equal confidence in results.

2. Breadth of insights

Advanced techniques let you test up to 3-dozen variables at once.  If it's true that "the more you test, the more you learn," you now have the tools to learn 10-times faster than you ever could before.

In that first online multivariable test—a large banner ad test—only one-third of the elements had any impact.  By testing many ideas at once, you can quickly determine (as John Wanamaker said) "which half of your advertising is wasted."

3. Depth of insights

The complex mathematics underlying scientific tests allow you to change many variables at once, but in an organized way, so you separate out the impact of each change on its own, plus analyze interactions between variables.

The Conde Nast e-mail test (mentioned above) pinpointed a number of profitable changes.  Results also uncovered an important two-way interaction in the presentation of the offer: each change on its own was valuable, but both together were unnecessary.  This gave them the flexibility to alternate among a number of successful tactics without hurting conversion (see the slides for more details).

4. Accuracy

More efficient test designs mean greater statistical power and accuracy. 

A 4-element test of credit card rates and fees quantified the impact of each change alone and interactions produced when price points were changed in combination.  This led to an optimization model (with 6 effects) to calculate the most profitable balance of response rate, risk, and revenue.  (This case study was published in the International Journal of Research in Marketing.)

5. Profit

New knowledge is great, but the true measure of testing success is sales and profit growth.  All hype and hubris goes away when you quantify your ROI on testing.  It's simple: run your multivariable test (or two or three), create the winning combination and test it against your original "control," measure the improvement and calculate how much it's worth. 

Getting Started

With a large number of vendors and software platforms, you have more choices for Internet testing than in other channels.  You also have a confusing array of terms like: multivariable or multivariate testing, Taguchi methods, MVT, optimal designs, and experimental design (DOE).  One big challenge is understanding the difference among approaches so you can find the best solution for your unique channels and challenges.

There are no proprietary statistical techniques but there are specialized skills required to apply them successfully

Scientific testing is a result of decades of research by Ph.D. academics who have actively published new techniques in academic journals since the 1920s.  Though the underlying mathematics and statistics are complicated, no one "owns" any special testing technique.  That said, different practitioners use different test designs with varying levels of statistical rigor and skill. 

The challenge is to find a "test guide" who...
(a) Is well versed in a wide range of efficient test designs
(b) Can effectively translate textbook statistics to real-world application
(c) Has the marketing experience to integrate the statistical structure within your unique, dynamic marketing programs
(d) Can clearly explain the pros and cons of her techniques and strategies.

Decide how much you can do yourself and where you need guidance

The right level of support will help you minimize cost and resource constraints, maximize insights and ROI, and build valuable internal experience in the testing process. 

Statistical guidance
Unless you've read the statistical textbooks of Ledolter & Swersey, Montgomery, and Box, have applied the techniques in similar companies and channels, and have found a good strategy for streamlining academic techniques for front-line marketers, you should probably find someone to guide your team through the process, at least at first.

Support in execution
How flexible are your current systems to support multiple versions of e-mails, ads, or webpages?  A simple website test may require a dozen versions of a landing page to be served at random with each visitor tagged so data can be collected for each page version.  A more complex order process test may include numerous versions and combinations of 5-10 different pages.

If you cannot manage the execution of website tests in-house, then partnering with a vendor like Optimost or Offermatica (among others) may be helpful.  Both of these firms have strong software-based platforms that manage the complexity of test execution through a seamless connection to your website.  The Google Website Optimizer is a free option for companies with a Google AdWords account. 

Interestingly, these three firms (and others) tend to use different statistical test designs and analytical techniques.  This points out one challenge of multivariable testing—with a wide array of statistical techniques, methods for selecting and defining test elements, and strategies for executing tests, you have a large number of choices, but need to ask enough questions to find the right match for you.

If you would like more information, feel free to contact us with any questions and to discuss your unique marketing challenges.  (And by the way, if you came to this page through a paid Google Ad, then you're part of an 8-recipe fractional-factorial SEM test.)

Back to top