Concentrate your brainpower – Focus on the best ideas to test

Scientific discipline begins when you harness the power of your ideas and concentrate the brainstorm list down to a manageable number of your highest-potential ideas. Then you can group similar ideas to distill a shorter list of the most potent changes, polish the raw ideas into clear “testable” elements, define bold levels, and assess the mix of variables to maximize your potential return while minimizing time, cost and effort.

This step begins by grouping related ideas and clarify confusing wording. Also, similar ideas may be combined and broad ideas split into multiple variables. In a direct mail test, ideas may be grouped by components: the outer envelope, letter, inserts, price and offer. Then within these groups, similar ideas can be grouped, like all variables related to envelope size, shape, color, and window size.

Trim the list

The next step is to use some objective criteria to trim the list. We start with “RIABC criteria,” keeping elements within each group that are relevant, can be changed independently, and are actionable, bold, and cost-effective.

• Relevant = the idea fits within the scope of the test project
  Brainstorming ideas for an in-store packaged goods test, all media advertising ideas may be eliminated as outside the scope of the test.
   
• Independent = the variable can be changed back and forth no matter how other variables are set
  For a catalog cover test, the two ideas: “test free shipping & handling” and “test lower S&H charge” are not independent; you can test: current charge v. free, or lower v. higher S&H, but you cannot test free S&H and a lower charge for S&H simultaneously. In this case, you can combine both ideas into one variable.

Independence is important from both a marketing and statistical standpoint, to be sure you’re testing what you truly intended to test and in order to meet the statistical assumptions of the selected test design. If test elements are not independent, then they should be redefined, combined, or eliminated, or the test design should be adjusted accordingly.

One catalog test included the variable, “additional message on cover.” But when the test covers were designed, the additional message was shifted from top to bottom depending on the cover photo and whether or not a dot-whack was also present. This resulted in a 3-way interaction between the three elements. Instead of testing the value of the additional message (as intended), the inconsistent execution wound up testing a mixture of the message, where it was placed, and the cover layout. Fortunately, the interaction showed the best combination of all three elements, but missed the independent effect of the message by itself.

• Actionable = the idea can be implemented if it works
  —  If it works, you’ll do it
  —  The results of this one test can be generalized to other campaigns

One example of a variable that may not be actionable is a catalog cover photo. Even if one picture pulls better than another, you will never use the same picture on every future catalog. To be useful, the pictures should be different concepts—perhaps “image photo” versus “best-seller”—where the results can be generalized, even if the exact same variable will not be used forever more.

• Bold = a substantial change between levels in the test
  The natural variation of the marketplace creates a hurdle for every test. Unless a test element can shift sales beyond the noise level, then it will not show any significant effect. Two ways to overcome this uncertainty is to have a powerful test design with a large sample size and/or to test bold changes. The larger the effect, the smaller sample size you need to see a statistically-significant difference. Sample size calculations can help (see Sample size for the equation), but it’s also important to test major changes before testing minor tweaks.

Test bold changes first. If a bold change makes little difference, then tweaks will certainly have no impact. If a bold change proves to have a large impact, then smaller changes can be tested until you reach the optimal settings.

A magazine publisher tested elements of a direct mail campaign for new subscriptions, including a 3x increase in subscription price. After quantifying the impact of this bold change, they could better select other price points to test with a greater chance of increasing profit without hurting response.

• Cost-effective = test no-cost or low-cost elements before testing expensive changes
  Search for creative ways to increase marketing effectiveness without increasing costs before testing expensive new changes. Often a few creative changes each have minimal impact, but in combination give an impressive return. If these are zero-cost changes, then the implementation decision is a no-brainer. When the element increases costs along with sales, then more in-depth sensitivity analysis is required to ensure the improvement is worth it. This often raises more questions and slows implementation.

Sometimes an increase in marketing costs leads to an impressive jump in ROI—like with an additional package insert, colorful envelope, bright sticker, or new in-store display. These costly ideas are worth testing, but only after testing zero-cost changes. Plus high-cost variables should often be tested separately to reduce testing costs.

Back to top

Define test elements

After concentrating your list ideas and focusing on the most powerful and testable ideas, the next step is to define specific test elements from every idea. This requires careful selection of the test element and levels you want to test.

• Test Element = definition of the variable you will test
  For example, the idea “brighter envelope” needs to be clarified. Will the test element be “envelope color,” “color graphic on envelope,” “sticker on envelope,” “gold foil stripe,” or something else.

How the test elements are defined can ultimately determine the success or failure of a test. Are they powerful ideas, clearly defined, or are they vague and weak? What do you expect to learn from this element and what will you do with these insights? Your decision should leverage the team’s marketing experience and knowledge. Focus on what you think (but don’t yet know), what clues you’ve gotten from previous campaigns and/or other industry players, and where you believe the unrealized potential lies.

No PhD statistician or sophisticated test design can uncover powerful results from weak test elements. You can’t test everything at once, but you can concentrate your highest-potential ideas into a few—or few dozen—clear, concise test elements that will lead you to new insights and new opportunities for increasing sales and marketing effectiveness.

• Levels = the test settings for each element
  Most tests compare the current best level, or "control," against a new idea, or test level. Depending on your objectives and the selected test design, you may test multiple levels for some elements. However, if you plan to test a dozen or more elements, then selecting just two levels for each is most efficient. (Look at case studies like the E-mail or magazine price test for examples of how to test multiple levels.)

When testing two levels for each element, one level is often the control and the other is a bold change that someone thinks will improve performance. For example, if “envelope color” is the test element and the control level is a plain white envelope, then the new level may be “bright blue,” “gold foil stripe,” “4-color,” or any other change chosen by the marketing team. You could test five (or 55) levels like: white versus blue, green, pink, and black. But instead of wasting a large portion of your test on just one variable—that may not have any impact at all—it’s best to select one bold change to see how much greater impact you can achieve by adding color. Choose one bold, but reasonable, color. Then if it has a big impact, you can test other colors or designs.

One of the biggest challenges of marketing testing is the nearly endless range of levels you can test. The selection of elements and levels remains a combination of both art and science. You will never quantify the impact of every possible change, so you need to make an intelligent call as to what is most important to learn in each test. This is where marketers (and consultants) can use the same basic statistical techniques and end up with vastly different results.

This is also why the concept of a “test cycle” is so important. Instead of testing every possible level at once, test the biggest, boldest changes first. Then select new elements and levels based on the significant results from the first test. Continue to refine on the most important elements and start a new test cycle—larger followed by smaller tests—as your marketing programs, competitive landscape, and opportunities change.

Successful tests require the careful balance of diverse marketing and statistical demands. You need to cast a wide net for bold new ideas while focusing on the details of the marketing campaign. You need to give your team the freedom to break the rules followed by a disciplined approach to define specific test elements and fit the statistical framework within your marketing program. This up-front work ultimately has the greatest impact on the success or failure of every test.

Back to top

Assess priority, density and intensity

With your initial list of test elements and levels, take a step back and assess what you have.

• Priority of test elements – Review your list against the original list of ideas. Have the highest-potential ideas remained, or has the list been watered-down as you defined test elements? Are these the best variables to test based on your marketing objectives?
   
• Density within the marketing program – Review the major components/areas of your marketing program to ensure the test touches on all important areas.

If you’re testing a catalog cover, do you have test elements on the back cover and front cover, inside and out, messages, offers, and graphics? Or if you want to test an outer envelope, be sure you’re testing all parts of the envelope, since sample size is independent of the number of variables in the test.

• Intensity of levels – Keep pushing for bold, yet reasonable, levels. The statistical concept of “convergence to the norm” points out the natural tendency for people to revert back to average performance. This means that, without close attention, big new ideas often become watered-down while safe, incremental changes prevail. Incremental changes are fine if your objective is to refine the levels of a few key variables. But if you want to break new ground, then you need to keep pushing your team beyond their comfort zone. Certainly, there’s also a point where bold levels become unreasonable, so you need to dig for changes close to the edge, without falling over.

An Internet retailer wanted to test more products on their landing page, so they selected 3 products versus 12 products as the two levels for the test. More than 12 products might be bolder, but could also confuse customers, slow the page loading, and be too difficult to manage from an operational standpoint.

Another e-commerce firm wanted to test about a dozen different messages in a banner ad. They were concerned that every test cell with more than two or three messages would kill response—no matter what the messages stated—so they were careful to create a test that avoided a mess of confusing elements all at once.

Test elements determine your maximum potential—how much improvement is possible if you accurately measure the impact of every variable (and combination) in the test. The test design, sample size, back-end analyses, and other statistics that follow help you see the reality of the marketplace more clearly, but they cannot uncover elements or relationships that you did not test. So take the time to cast a wide net for new ideas, then focus on strong test elements.

Learn more about selecting key metrics. Then move along to the next step: creating a test design with impressive statistical power.

Back to top