Like trading in your slide rule for a supercomputer…

		Test dozens of marketing variables as fast as one alone (test 10-20x more variables than you ever could before)
		Use the same sample size whether you’re testing one or 31 elements (often just 5% to 10% of the sample size normally required)
		Accurately quantify the real-world impact of each element on its own, plus interactions between test elements (optimize every detail at once instead of a broad “test-control” approach)
		Use the right test design to meet your marketing objectives (flexible, efficient designs let you increase your ROI from every test)

Numerous specialized statistical techniques are used in the test design phase. With myriad options for testing any number of variables and levels in different combinations, this step requires a good understanding of the unique statistics of test design (see scientific testing for the technical details).

After selecting and defining your test elements and levels, you can then create the statistical test design, showing the specific combinations of test elements, or “recipes” (test cells), that must be created for a valid multivariable scientific test. The case studies & articles show numerous examples of test designs for direct marketing, Internet, and retail applications.

Four key steps include:

1. Plan an efficient test strategy

The test strategy should outline the scope, schedule, resources, and constraints for the test. Often the test strategy includes a series of “screening” and “refining” tests, followed by the final implementation drop, so one project may cover a number of tests or mail drops.

2. Select a powerful test design

This step begins even as you trim the list and define test elements and levels. The test design should be based on your objectives, available resources, and design alternatives.

• Objectives – number of variables, number of levels, importance of main effects and interactions
• Resources – speed, sample size, number of test cells possible, cost of testing
• Design alternatives – statistical requirements for each test

3. Create test recipes

The test design will dictate which combinations of all the variables need to be tested, with some flexibility to avoid undesirable combinations. Along with the definition of test elements, the creation of test recipes is by far the most important step of the process. The test expert should provide your marketing team with simple “recipe sheets” listing how each variable must be set in every recipe.

It’s very important in this step to:

• Create test elements and levels before creating recipes – so elements remain independent, bold, consistent, and easy to combine
• Review a “mental matrix” before developing the physical recipes – think through challenges and potential interactions before starting
• Change elements, levels, and recipes if necessary – be sure elements and recipes remain bold yet reasonable
• Create all final test recipes – be certain the test expert reviews all creative and production proofs

4. Launch the test

This final step may be as easy as sending out a catalog or webpage, or as challenging as monitoring every element in a retail test each week in 50 (or 500) different stores. With direct marketing, the effort and attention comes before the test is launched, whereas packaged goods, magazine newsstand, and other retail tests require on-going support until the test is over.
 

Testing is as much art as science

Just as each brushstroke adds to a painting, every detail of the test strategy adds to—or detracts from—the beauty of your results. As the fluid process of testing takes shape over time, your potential insights begin to solidify. Yet with skill and experience, careful attention to detail, and solid execution, the reality of your marketplace will reveal itself in vivid detail.

From here, the next step is the most exciting part: uncovering marketing insights from your test results. You can also look over some real-world case studies & articles, or you can learn more about the details of scientific testing, explaining how you can test many variables at once.

Back to top