The art and science of market testing

Editor's note: Gordon H. Bell is the principal of Montlake Marketing, a Knoxville, Tenn., consulting firm.

After 75 years, market testing is becoming the Cinderella of the research world. Initially, either ignored or over-worked, testing was often perceived as too simple or unglamorous to be worth the attention of high-level managers. Also, with only basic tools and minimal education in the testing process, many marketers had a superficial understanding of how real-world tests should be designed and managed. So testing has been crude and hindered by unclear results.

Now the transformation has begun. A new, streamlined approach has given marketers a clear strategy to harness creativity and identify new opportunities for growth. In addition, advanced statistical techniques - first introduced nearly 50 years ago - have recently been refined, offering managers the freedom to test dozens of elements all at once as fast as most companies test one new idea alone.

The matrix marketing strategy: raw ideas transformed to a wealth of opportunity

The ultimate "glass slipper" is a strategy that fits the art together with the science of market testing. It combines creative freedom and scientific discipline to help marketers brainstorm new ideas, identify a solid list of variables that may impact sales, run a fast multivariable market test, quantify the precise impact of each test element, and rapidly implement changes proven to increase profitability. This new approach, often called matrix marketing (MxM), helps marketers pinpoint ways to rapidly increase sales through direct mail, catalog, retail, advertising, e-mail, and Internet channels.

The creative benefit of matrix marketing is its ability to quickly pinpoint the best ideas to improve market performance. The technical benefits of MxM come down to quantity, speed, and depth of learning. Marketers can test a multitude of creative, offer, list, and product elements all in one market test. They can test three or 33 variables with the same sample size just as fast as a test of one new creative against the "control." In addition, results show exactly why a new creative worked - ranking the elements from best to worst, pinpointing the root cause of good results, and giving in-depth information about the impact of each element on its own and in combination with others.

The basic matrix marketing strategy is shown in Figure 1.

The first step is to assess marketing programs and metrics to find the best opportunity to increase sales and strengthen competitive position. The initial MxM project should have:

• clear measures of success (objective, quantifiable data);
• strategic significance and high potential payback;
• visible senior management support;
• production flexibility (multiple versions possible);
• speed (minimal development and data lag).

The second step is to pull everyone together and brainstorm every possible change that may increase sales or reduce costs: segmentation, positioning, offers, vehicles, words, pictures, or any other specific change someone thinks may help.

Next, trim the list to one to three dozen elemental ideas, ideas that can be changed independently and are actionable, bold, and cost-effective:

• independent - each element can be changed without affecting the others;
• actionable - all elements can be implemented quickly and safely, with generalizable results that can be broadly applied to future campaigns;
• bold - substantial change between test levels (avoid testing "tweaks");
• cost-effective - the change should cost no more than current tactics.

From this list, the marketing team can then define specific test elements with two levels for each: the current "control" and a change that someone thinks will improve marketing effectiveness. For example, "Make the envelope stand out in the mail" may be an idea; "envelope color: white versus bright blue" may become the test element.

These first four steps of the matrix marketing strategy require no statistics, but are the most important part of any test. For every test element, you must ask: "What can we learn from this?" and "Is the change bold enough to stretch our market knowledge?" Matrix marketing should leverage the marketing team's skills and experience while pushing them beyond their comfort zone.

The statistical power of this multivariable test strategy is concentrated within the "screening" and "refining" tests. Screening tests quantify the impact of lots of variables at once, while refining tests zero in on the best possible settings for the few important elements. This screen-refine test cycle lets you cast a wide net to search for new opportunities, then quickly optimize and implement the best changes to increase sales and reduce costs. After interpreting and implementing results - building market wisdom along with profit - the team can move on to the next opportunity.

Case study: e-mail test

Now we'll look at an example of an MxM screening test. A marketing vice president wanted to increase the effectiveness of her e-mail marketing programs - increase open rate, click-through, and sales, plus reduce unsubscribes. Testing HTML e-mails, she also hoped to find creative elements that could be confirmed and implemented on the company Web site. Her team brainstormed 95 ideas, then zeroed in on 18 elements to test - all at once, in one drop, with 150,000 names (Figure 2).

An expert worked with the team to design a matrix to test everything with just 20 e-mail "recipes," each with a unique combination of all 18 elements. Though the statistical basis of these test designs gets complicated, the concept is fairly straightforward.

Matrix tests require at least one more recipe than the number of test elements and the number of recipes is always some multiple of four. For example, marketers can test 15 variables with 16 recipes, 16-19 variables with only 20 test recipes, up to 23 variables with 24 recipes, and so on. Every recipe gives information about every test element, so sample size is not affected by the number of test elements or recipes. In fact, if a simple "champion-challenger" test with two e-mail versions requires at least 150,000 names (for statistical precision), then with those same 150,000 names you can test up to 39 different variables all at once.

In addition, since the matrix marketing strategy tests combinations of elements, you can also calculate interactions between elements. For example, one test showed that a free gift and 15 percent discount each increased sales, but both together were no better than the discount alone. Another test showed that a sticker on the letter had a much larger impact combined with a low product price than with a high price. Figure 3 shows the MxM matrix for the e-mail test.

How to create the "recipes" in a matrix

Looking at the matrix, each row is a recipe - one version of the e-mail. Each column is one test element (A-R). Every minus in the matrix means that element should be set as in the control and every plus means the element is changed to the "new idea."

First look at recipe No. 20: this row is all-minus, or the control. Each other recipe has about half the elements like the control and half with the new setting. Now look at recipe No. 1. To create this version, you need to:

• send it only to previous responders over 35 (A+, B+);
• use the "special offer" subject line with no personalization (C-, D-);
• have the sender name a person (E+);
• use the new page design with graphics on the sidebar, brighter colors and no animation (F+, G+, H+, I-);
• insert a table of contents, use long, direct-sell copy, no section headings, and one link per offer (J+, K-, L+, M, N-);
• do not offer the special discount or 800-number (O-, P-);
• use bold and color in the font and the new unsubscribe copy (Q+, R+).

Then for recipe No. 2, the e-mail will be sent to a different segment and eight other elements must be changed from recipe No. 1.

The process does take a bit more effort than a simple A/B split, but the return - in knowledge and profit - more than makes up for a few extra days of creative work. Also, the independence of test elements means that after all elements and +/- levels are defined, the recipes are simply cut-and-paste combinations.

The basic math behind the high-powered statistics

These test designs are based on matrix algebra, statistical probability, and more advanced mathematical concepts. However, in simple terms, the reason this type of test design is so efficient is the perfect balance of the + and - levels. For example, look at column A: you see 10 + and 10 - recipes. Now look at just the 10 + recipes (1, 3, 4, 9, 11, 12, 13, 15, 18, 19): column B has five + and five -; column C has five + and -; columns D through R are similar. With this perfect balance, you can calculate the effect of each test element independently of everything else you change. The effect equals the average of all + recipes minus the average of all - recipes. No matter how large or small the effect of all other variables, the impact of element A comes out the same.

The analysis is a bit more complex due to experimental error, confounding error, and the unique challenges of dynamic markets and uncontrolled variation. Yet the basics remain the same - with a solid group of elements, good test design, and careful supervision, a well-run test should give fast, clear, cause-and-effect results.

This balance (or "orthogonality") also means that you can use the same sample size no matter how many variables you test. If you normally need to mail 300,000 catalogs for a simple A/B split, you can mail the same number for a 24-recipe test - just 12,500 books per recipe.

Testing all 18 of these e-mail elements using simple champion-challenger test methods would require 1,425,000 names (essentially, 18 separate tests). Alternately, testing all combinations of these 18 elements would require 262,144 recipes. Matrix marketing provides all the real-world data you need to prove what sells, but with a manageable number of names and test recipes. The key is to include all variables within one test, with an element-to-recipe ratio of between 1:1 and 1:2 (the higher ratio when advanced techniques are used to give more detailed information).

Many marketers see such a large return on MxM because they never before had the creative freedom and scientific precision to pinpoint the impact of so many new ideas at once. So instead of finding one or two changes to increase sales, they discover five or 10 all at once.

For the e-mail test, all 20 versions were sent on the same day. Test results, analyzed in real time over hours and days, showed the precise impact of each element on its own and interactions among elements. Figure 4 summarizes what they found.

Test results

Looking at Figure 4, everything above the dashed line is statistically significant. On the left is the list of all "new ideas," showing their impact from largest (A) to smallest (Q). On the right, the bar shows the size of each effect - as a percentage change in sales produced by switching from the minus to the plus level - and whether the "new idea" was better (helps) or worse (hurts) than the control.

For example, element A ("previous responders") helps increase sales by 27 percent (it was added, along with "age," to segment the list and block known differences from the experimental error). This means that previous responders bought 27 percent more product on average than non-previous responders. Then for K ("copy - informative"), the "new idea" actually reduced sales by 21 percent versus the control of "more direct sell"; people who received the e-mails with informative copy bought 21 percent less product, on average, than those who received the direct-sell copy.

Based on these market results, the best e-mail will increase sales 73 percent over the control (without including the sales increase from "previous responders"). Many of these important elements - formats, graphics, content, and offers - were also confirmed and implemented on the Web site.

Overall, sales increased with the new page design, table of contents, 800-number, and special discount. Personalization and section headings in the copy also helped. Some new ideas hurt sales: more informative copy, a creative subject, longer copy, and sidebar graphics all reduced sales, so the team kept these at the control level. In addition, elements that made no difference were adjusted to reduce file size.

If all new ideas had been implemented after a typical "package" test, they would have seen an 11 percent increase in sales - not bad, but a big difference from the 73 percent gain they achieved by implementing the six helpful and avoiding the four harmful elements.

Interactions

Matrix marketing can also uncover interactions among test elements. In this case, copy message (K) and copy length (L) produced an important two-way interaction. Though overall, the direct-sell message and short copy increased sales, the long copy was actually slightly better when the more informative message was used. This did not alter their final decisions, but did provide information about the relationship between copy content and length.

After the screening matrix is completed, one or two smaller refining tests often follow. A refining matrix follows the same principles as screening, but is designed to better clarify interactions and also test new combinations of significant elements. Within the matrix marketing strategy, you can use a wide range of multivariable test designs depending on your interests - whether testing a multitude of new ideas quickly, or refining a few important elements to optimize performance.

Creative freedom + scientific precision = rapid growth

How many great marketing efforts fall flat in the marketplace? The answer is not to restrict new ideas, but to test and quantify the impact of everything you want to try. Marketers who do it right are like creative scientists...generating a burst of creative energy, harnessing the power, and focusing it on bottom-line results.

Matrix marketing gives you the freedom to simultaneously test dozens of innovative ideas, zero in on the specific tactics and elements that increase sales, and rapidly implement proven changes. The power of matrix marketing is no fairy tale, but a way to quickly transform simple ideas into genuine success in the marketplace.