Questions:

1.  Honestly, I've been testing for years. What can a consultant tell me that I don't already know?
2.  If this stuff is so good, why haven't I heard of it before?
3.  I’m hearing more about experimental design, Taguchi methods, multivariable testing, matrix marketing, Six Sigma, and a pile of other new-and-improved testing methods. What’s the difference?
4.  Some big firms have begun doing this stuff, so why is LucidView anything special?
5.  Scientific testing sounds complicated. I know testing is important, but we don't have time for something new. Really, how much better is scientific testing than split-run techniques?
6.  Can’t our statistician do this herself?
7.  We have no budget for testing. What, if anything, can we still do?
8.  What range of improvement can I expect from a well-run test?
 

1.  Honestly, I've been testing for years. What can a consultant tell me that I don't already know?
If you've been testing for years, you're in a good position to implement advanced scientific techniques. With testing programs already in place, you have the skills and mindset to leverage the power of cutting-edge techniques. But scientific testing requires statistical know-how that cannot be gained by experience alone. Learn more about the details before jumping to conclusions.

2.  If this stuff is so good, why haven't I heard of it before?
The field of scientific testing is like an advanced civilization lost in the jungle. Statistical experts largely ignored the marketing world while one-variable-at-a-time testing became more firmly entrenched among market leaders. It’s as if business executives have been driving Model T Fords without ever seeing the Ferrari around the next block. Plus there’s a large gap between statistical theory and in-market application. As with many new technologies, it’s taken years for advanced techniques to be refined and streamlined for real-world marketing tests. In addition, the most successful companies have worked diligently to hide the secret of their success. And finally, after learning the “scientific method” in elementary school, the epiphany can be hard to handle.

Back to top

3.  I’m hearing more about experimental design, Taguchi methods, multivariable testing, matrix marketing, Six Sigma, and a pile of other new-and-improved testing methods. What’s the difference?
In a word: branding. With most statistical advances coming from academia, cutting-edge techniques are readily available to anyone who can dig through the textbooks and journal articles and uncover the useful information buried within the complexity. No one owns some magical proprietary statistical technique that you need to have. And many of these advanced techniques have been around for decades.

The difference is (a) what you call it, and (b) how you use it. Terminology aside, the statistical techniques are tools. One expert or another may have a slightly different collection of tools in their toolbox, but anyone can find a hammer or a wrench. The difference is in the application of the various tools. Just as your experience with a hammer does not mean you can build a house, knowledge of statistics does not mean you can achieve immense market success. With a large gap between academic theory and in-market application, experience and hands-on expertise are imperative.

4.  Some big firms have begun doing this stuff, so why is LucidView anything special?
Three words: experience, expertise, focus. After testing thousands of variables in hundreds of tests for dozens of market leaders, LucidView consultants have made a career of helping market leaders improve performance through scientific testing. By focusing on scientific marketing testing, LucidView has refined the techniques and streamlined an approach for the unique requirements of in-market testing. You'll be hard-pressed to find any other group who has worked with so many academic experts and industry leaders.

That said, certainly we are not the only people on earth doing this stuff. You should take some time to learn about the various companies and strategies available. In your search consider a few points:
(a) One person will (should) guide you through the scientific testing process from start-to-finish, so find the person who you believe has the unique skills and experience to work with your team and achieve profitable results.
(b) Scientific testing is a specialized area within the vast field of statistics. Just as a copywriter and graphic designer are not interchangeable, statisticians with skills in modeling, data-mining, and other analytical tools are much different than experts in testing. Find someone who has done a lot of testing and done it well.
(c) The testing strategy should be easy to integrate within your on-going marketing programs. Find someone who can work hands-on with your team to reduce the learning curve, speed results, and help you learn the techniques.
(d) Question the details. Ask for examples of successful and unsuccessful tests and the cause for both. Ask if impressive numbers are based on implemented results or predicted results. Talk directly with the people who led the marketing projects to be sure they understand the unique requirements of your industry and marketing programs.

Back to top

5.  Scientific testing sounds complicated. I know testing is important, but we don't have time for something new. Really, how much better is scientific testing than split-run techniques?
You have two questions here: (1) how complicated is it; (2) is it really worth the effort. #1: The techniques are complex, but with an experienced guide, the complexity can remain transparent to your marketing team. Plan to add a few days to your creative cycle, but don't think you have to become a statistical expert. Ultimately, scientific testing gives you greater creative freedom to test more ideas more quickly. Find someone who can worry about the statistics, so you don't have to you.

#2: Is it worth the effort? Yes. If you use is strategically—selecting the right project and appropriate test design—then scientific testing lets you test 10-20 times more variables in a fraction of the time. Think about it this way: maybe 1/4th of your new ideas will actually improve performance. If you test four variables against your control in one mail drop, you may find one change that helps. Now with scientific testing, you can test two-dozen variables with a smaller sample size that you need for those four split-run tests. With 24 test elements, you may find 6 changes to improve performance—six times more than you would have learned before.

6.  Can’t our statistician do this herself?
With lots of experience, some additional training, and good management skills, yes. Remember that the statistics behind scientific testing comprise a small, specialized field within the vast realm of statistical concepts and techniques. Most statistical programs in universities include one course on “experimental design,” using a textbook that never once mentions the word “marketing” or “advertising.” Courses often cover matrix algebra and the derivation of different test designs (an interesting math lesson, but of little real-world value), without any case studies beyond manufacturing and engineering applications. So, would you hire a marketing executive after one college course and no experience?

Actually, your statisticians have many of the core skills required for successful testing, but they can benefit greatly from additional training and hands-on experience. You have two main choices: (1) flounder through it yourself and hope the learning curve is not too slow, or (2) find an experienced guide to help you hit the ground running, increase ROI, and teach you the most efficient way to test. And honestly, a good consultant will answer any questions you have. Absorb all the knowledge you can as quickly as possible and then you can decide for yourself when your team is ready to do it themselves.

Back to top

7.  We have no budget for testing. What, if anything, can we still do?
Testing is an investment in your future success. Companies that don’t test can, at best, slow their rate of decline. You need to test. You may have no “budget” for testing, but think about how much you spend on every marketing campaign? Testing is an integral part of marketing, so some portion of your marketing budget must be used for testing. Consider the upside potential: if you increase response by 10% or 50% or 100%, what is that worth? How much would you invest to receive a 50% jump in response after your next drop?

Certainly costs are an issue. That’s why you may want to test changes to only the outer envelope, or test an e-mail campaign before a catalog cover, or run a test with only your top customer segments. But as long as you keep mailing, you better keep testing. Fortunately, scientific testing lets you do it faster, more efficiently, and with a bigger potential for success.

8.  What range of improvement can I expect from a well-run test?
Good question. Unfortunately, there’s no easy answer. I’ve seen tests that eke out a measly 5% lift (which may be big dollars for a large, mature program) and I’ve seen tests that give a 330% lift. One big difference is often in the test elements. If your team comes up with a few “home run” ideas, then the improvement can be impressive. But even if your ideas each have minimal impact, by testing many ideas at once, you can often achieve an impressive lift. In addition, a good consultant can squeeze out the best results from every step of the testing process.

Please call or e-mail us with any other questions you may have.

Back to top