Editor’s note: Gordon Bell is president of LucidView, an Oak Ridge, Tenn., consulting firm. Roger Longbotham is a senior statistician with Seattle-based Amazon.com.

Multivariable testing in marketing is like the gold rush of the 1800s. New “discoveries” hit the press and we rush off to mine the next breakthrough technique. But the reality is not quite so glamorous - or chaotic. This “new” field of multivariable testing is actually the result of decades of academic research and statistical practice, with impressive depth beyond the basic terms and concepts that reach the marketing press.

In testing - as in marketing - clarity and efficiency should take precedence over technical showmanship. Statistical complexity on its own has little inherent value unless it achieves an obvious increase in ROI. The key is to find the right balance between powerful statistics, a user-friendly approach and clear, actionable results.

Efficient and flexible

Some multivariable test designs are both powerful and easy to understand. Full-factorial, fractional-factorial and Placket-Burman designs provide a solid foundation for efficient and flexible multivariable testing in marketing. You can use versions of these to test two or two dozen variables, analyze main effects alone or in combination with interactions and adjust the size and layout of the test design to meet your marketing objectives and constraints.

Other statistical designs sacrifice ease of use in order to achieve a specialized objective. Computer-generated “optimal” test designs are one example. First developed in the late 1950s for manufacturing experiments, optimal designs offer a way to run experiments under non-standard conditions. For example, in a manufacturing test of machine speed and flow rate, the combination of high speed and low flow may burn out the machine, so this combination must be avoided. Optimal designs allow you to test under sub-optimal conditions where certain combinations are constrained, the cost of testing is immense or the “response surface” has abnormal characteristics. Fortunately, these constraints are seldom necessary in marketing tests.

The D-optimality criterion is one method for defining optimal designs. For this approach, a design is set up to minimize the volume of the confidence region of the effect estimates (considering the variances and covariances of these estimates). Other optimality criteria will result in different test designs. In addition to D-optimality, statisticians have defined A-, C-, E-, G-, I- and S- (and other) optimality criteria. Even the same optimality criterion may result in different test designs depending on the optimization software. Simply put, if none of this paragraph makes sense to you, then these optimal test designs become a “black box.” If you cannot use your marketing experience to interpret the results, then the statistical output should be implemented with great care.

In addition, optimal designs create a number of challenges:

  • These designs are applicable when your test variables are continuous (like temperature and pressure). However, when you have discrete variables, as we normally do in marketing tests, they either don’t work or provide little or no benefit.
  • The computer creates a design based solely on the input criteria and the underlying assumptions implicit in the approach. Optimal designs generally assume some form for the model, relationship or range of influence for the variables being studied. If these assumptions are not met, the design is no longer optimal. Rarely do we know this much about the relationships prior to conducting the test - that’s why we are doing it!
  • The complex analyses require advanced statistical skills. Unless (or even if) you have a Ph.D. statistician on staff, the analysis can be challenging and the results can be very confusing.
  • Results are not only difficult to interpret but may change based on the selected criteria and assumed effects. Forcing constraints is like removing boards from the framing of a house - a few changes may be OK, but you never quite know when you have weakened the structure too much.
  • The small increase in statistical power comes with a large increase in complexity. These designs make great journal articles but are not very practical for most real-world applications.

In multivariable marketing testing, the most “optimal” test design is usually one with a straightforward execution, clear analysis and easily understood results. From both a marketing and statistical perspective, esoteric designs like D-optimal designs are often a sub-optimal choice.

The right techniques at the right time

Multivariable testing is most effective as a strategic marketing tool. Test designs offer an efficient framework for testing your new ideas. But just as the framing of a house is only the first step towards a beautiful home, what you place upon the statistical framework is what ultimately determines the attractiveness of your test results.

Strategic testing means using the right techniques at the right time. What is your biggest opportunity to increase marketing ROI? What are the primary questions you want to answer in each test? Once you answer those questions, then you can follow a logical, structured approach:

1. Plan a series of tests. One test cannot answer every question. Consider a cycle of testing, where you build upon results from each test and refocus your marketing programs as you gain new insights. You can test many creative elements to determine which are important, or refine price and offer variables to quantify key interactions, or test your contact strategy to pinpoint profitable touchpoints, but testing all of these together quickly becomes unmanageable.

2. Answer the big questions before fine-tuning your programs. Find out which marketing-mix elements are important before testing the details. For example, you can find out if envelope color makes a difference (perhaps testing a white versus blue envelope) before testing five different shades of color. Or you can quantify the impact of a 20 percent price increase, before testing 5 percent, 8 percent, 10 percent, 15 percent and 20 percent changes all at once.

This also means that two-level test elements are frequently more efficient than multilevel designs. Especially with creative elements, two levels can provide more useful and actionable information. For example, a test of two headlines, creative and offer-focused, can show a) if different headlines have a different impact and b) what type of headline is most effective. If the offer headline is more effective, then the next test can focus on different wording. In addition, multiple levels usually require larger sample size, more test cells and more complex analyses and create real difficulty in analyzing interactions.

3. Find the most powerful and efficient method for testing your ideas. The simplest solution is often the best. Real-world tests in dynamic markets with limited resources are much different than theoretical experiments in a controlled laboratory environment managed by Ph.D. statisticians. Putting powerful tools into the hands of marketers is more important than using the most theoretically advanced statistics. Testing bold new ideas, executing clean and fast marketing tests and rapidly improving performance is where the real power of multivariable techniques rests.

4. Understand the statistical rules you need to follow. The statistics encourage some self-restraint. Every test requires a balance between creative freedom and statistical structure. Part of the art of testing is finding a way to leverage your team’s brainpower within the statistical constraints required to achieve reliable test results.

Gain useful insights

When the test elements, execution and results and clear and understandable, the marketing team is much more likely to gain useful insights and implement the results. If the test is a black box with confusing data, then the results may never be understood or implemented.

Although the underlying statistical theory is daunting, every marketer can understand the basic pros and cons of each scientific test design they execute. Your guide should be able to explain the alternatives and why the selected test design is the optimal choice for your unique marketing program and objectives.

Multivariable testing is a powerful tool to help you learn more, faster, with fewer resources. Yet like every tool, how you use it is the key to success. With a full toolbox of test designs you can find the most efficient technique for each situation. When the statistics become transparent to your marketing team, you free their creative energy for explosive growth.