For a concise introduction, The Rising Tide in Circulation Testing, Navigating the Depths of Multivariable Testing, and the case study of Direct Mail Subscriptions Tests are a good place to start. Then look over other case studies, articles, and conference presentations, or contact us for more examples of direct mail, retail, Internet, catalog, advertising, and other marketing tests.

Note that terms used in the articles over the years—like e-testing, multivariable and multivariate testing, matrix testing, and matrix marketing—are variations on the theme of scientific testing.

Scroll down (or click below) to open:
» Published articles (full articles and reprints)
» Published articles (excerpts)
» 1-page Case Studies
» Detailed Case Studies (showing test elements, designs, and results)
» Conference Presentations (recent)

Published Articles - reprints and online

1.  The Rising Tide in Circulation Testing (online article)
      ●    A concise introduction to the benefits of scientific testing from CM magazine
      ●    With a comparison of split-run versus multivariable testing from a direct mail campaign

2.  Navigating the Depths of Multivariable Testing (article pdf)
      ●    Good background and introduction to scientific testing from Internet Retailer
      ●    With details of an 18-element e-mail test

3.  The Right Sample Size will bring Your Tests into Focus (online article)
      ●    Explanation and  from CM magazine of the benefits of scientific multivariable testing
      ●    Brief examples from the publishing industry of direct mail, Internet, and newsstand tests and how to get started

4.  Never Trust the Numbers (article pdf)
      ●    A humorous and useful look at where you can go wrong analyzing "objective" data

5.  A Scientific Approach to Implementing Change (article pdf)
      ●    A unique retail test of organizational change

6.  Experimental Design on the Front Lines of Marketing (online abstract)
      ●    A technical article with credit card direct mail tests to explain the theory and application of complex statistical testing techniques
      ●    Contact us for a reprint from the International Journal of Research in Marketing

7.  (Sub-) Optimal Test Designs for Multivariable Marketing Testing (article pdf)
      ●    Advanced article discussing pros and cons of optimal test designs
      ●    The importance of test strategy is also covered in this article from Quirk's Marketing Research Review

8.  Beyond the A/B Split: Cutting-edge Techniques in Test Design (article pdf)
      ●    Explains the basics of scientific testing, with a simple example
      ●    The Target Marketing article also shows a large e-mail test

9.  The Art and Science of Market Testing (online article)
      ●    A detailed discussion of benefits and strategy, with an e-mail example

10.  E-Testing in Internet Time (article pdf)
      ●    An early article explaining Internet testing
      ●    A unique test design combining e-mail, online ads, and webpages

Published Articles - excerpts

11. Dispelling Some Market Testing Myths
      ●    A brief overview of 6 common myths of marketing testing

12. Drive Marketing Results with Multivariable Testing
      ●    An early article with a good explanation of the benefits of scientific testing
      ●    Examples of catalog, retail, and telemarketing tests

13.  Multivariable Testing Methods
      ●    General overview with a print advertising test

14.  The Quest for Net Profit
      ●    An early article with a brief example of an Internet test

15. Physics and the Laws of Marketing
      ●    For something completely different... 3 laws of strategic marketing

Back to top

One-page Case Studies

16. Advertising Media Mix Test
      ●    Test of 10 media changes to increase retail sales
      ●    Results increased sales 8%, worth $26 million

17. Outbound Telemarketing Test
      ●    11 changes to the script and offer were tested simultaneously
      ●    Conversion rate tripled and one interaction increased sales 20%

18. Call Center Attrition Test
      ●    15 ideas to increase customer retention tested at once
      ●    Account attrition dropped 18% and transactions increased 12%

Detailed Case Studies - with explanations of test strategies and designs

19. Direct Mail Subscriptions Tests
      ●    Case study of a successful strategy for testing
      ●    11-element “screening” test and 5-element “refining” test
      ●    Response jumped 70% after implementing results

20. Retail Marketing Mix Test
      ●    Test of 10 in-store advertising, display, and promotional elements
      ●    Using 15% of the sample size required for champion-challenger tests
      ●    21% increase in retail sales

21. E-mail Office Supplies Test
      ●    Statistically detailed case study explaining test designs and analyses
      ●    Large test with a small sample size run across three market segments
      ●    Detailed discussion of interactions (and a 54% increase in conversion)

22. Landing Page Test
      ●    Concise case study of a fast 23-element test for a specialty-clothing marketer
      ●    Learned in 3 weeks what normally would require 15 months of testing
      ●    42% jump in conversion rate with 11 significant elements

23. Newsstand Price Test
      ●    Small test with big surprises
      ●    Three elements tested at three levels each
      ●    Valuable interactions among price elements

24. Catalog Cover and Insert Test
      ●    15 marketing elements tested in one drop
      ●    Sample size 90% smaller than for split-run tests
      ●    Response rate increased 18% after results were implemented

Back to top

Conference Presentations - contact us for details

25. Direct Mail Testing: Innovations and Insights for Challenging Markets
       ●    Click here for a pdf copy of the session slides from DMA07 (the Direct Marketing Association annual conference) in Chicago, October 16, 2007
       ●    The session covers tactics for successful split-run and multivariable testing, with case studies from THD and Financial Times
       ●    Speakers: Richard Varey, Financial Times; Chris Law, THD; and Gordon Bell, LucidView

26. Design of Experiments in Marketing and Advertising Testing
       ●    Contact us for a copy of the paper from the 2007 International Statistical Institute bi-annual conference (ISI 2007) in Lisbon, Portugal
       ●    Gordon Bell joined the international panel of speakers in the first-ever session on applications of advanced experimental design techniques (i.e., multivariable testing) in marketing, retail, and advertising (IPM 70)

27. 4-D Multivariable Testing to Optimize Contact Strategy
       ●    Contact us for slides from the 2007 DM Days - New York session on multivariable contact strategy testing in direct mail
       ●    Speakers included Jennifer Clark (Marketing Manager, Whirlpool), Ryan Mitcham (Database Manager, Accent), and Fred Holland (Vice President, LucidView)

28. Circulation Testing: Techniques and Strategies for Increasing Insights and ROI
       ●    Click here for a pdf copy of the session slides on multivariable and split-run catalog testing from the Annual Catalog Conference (ACCM), May 23, 2007
       ●    Speakers: John Buleza (VP of Marketing, Lillian Vernon) and Gordon Bell (President, LucidView)

29. Multivariable Testing: Secrets of Success from Conde Nast and Ameriprise
       ●    Click here to see the session slides from the 2006 Direct Marketing Association Annual Conference (DMA06), including case studies from:
       ●    Tammy R. Nelson, Director of Marketing, Ameriprise Auto & Home Insurance
       ●    Chet Van Wert, Strategic Marketing Director, Condé Nast Publications

30. MarketingSherpa E-mail Summit, April 20-21, 2006 in Chicago
      ●    Contact us for a copy of the "Advanced Testing 101" session slides

31. SRI Database Marketing Conference, March 23-24, 2006 in Seattle
       ●    Contact us for a copy of Gordon Bell's session slides on: 
       ●    "The Science of Testing," with Roger Longbotham, Senior Statistician, Amazon.com
       ●    "Experimental Design and The Art of Testing," with Tammy R. Nelson, Director of Marketing, Ameriprise Auto & Home Insurance

32. Sprint Nextel: The Bottom-line Impact of Effective Analytics
       ●    From the 2005 Direct Marketing Association Annual Conference (DMA05)
       ●    Click here to see the session slides

33. Direct Marketing Stats Bootcamp
       ●    From the 2005 Annual Catalog Conference
       ●    Contact us for a copy of the slides from the session

34. The Art and Science of Successful Direct Marketing Testing
       ●    From the 2003 Direct Marketing Association Annual Conference
       ●    Contact us for a CD of the full presentation

35. Innovations in Retail Testing and Measurement
       ●    From the 2003 Advertising Research Federation Annual Convention
       ●    Click here for a copy of the technical paper

After you read over a few articles and case studies, contact us about your specific advertising and marketing programs and how we can work together to increase your marketing ROI.

Back to top