Editor's note: Jerry Thomas is president of Decision Analyst Inc., Arlington, Texas.

Based on 30 years of marketing research experience that spans thousands of research projects, I am convinced that product testing is the single most valuable marketing research that most companies ever do.

The great value of product testing is perhaps best illustrated by its many uses. It can be used to:

  • achieve product superiority over competitive products;
  • continuously improve product performance and customer satisfaction (i.e., maintain product superiority, especially as consumer tastes evolve over time);
  • monitor the potential threat levels posed by competitive products and understand competitive strengths and weaknesses;
  • cost-reduce product formulations and/or processing methods, while maintaining product superiority;
  • measure the effects of aging on product quality (shelf-life studies);
  • implicitly measure the effects of brand, price, brand name or packaging on perceived product performance/quality;
  • provide guidance to research and development in developing new products or upgrading existing products;
  • monitor product quality from different factories, through different channels of distribution, from year to year; and
  • predict consumer acceptance of new products.

Companies committed to rigorous product testing and continuous product improvement can, in most instances, achieve product superiority over their competitors. Product superiority, in turn, helps strengthen brand share, magnifies the positive effects of all marketing activities (advertising, promotion, selling) and often allows the superior product to command a premium price relative to competitors.
Most companies, unfortunately, do very little product testing. Few companies really understand the power of continuous product improvement and product testing. Even fewer know how to do product testing the right way. Fewer still budget enough money to support a serious product-testing program. These shortcomings in most companies, of course, create opportunities for the few firms dedicated to continuous product improvement. How can companies realize optimal value from product testing?

Product testing secrets

There are several secrets to truly accurate and actionable product testing. They include:

  • A systems approach. The methods and procedures of product testing should constitute a standardized system, so that every like product is tested exactly the same way, including identical product preparation, age, packaging and coding; identical questionnaires (although parts of the questionnaire must be adapted to different product categories); identical sampling plans, typically using blocking-screening grids to ensure matched samples; identical data preparation and tabulation methods; and similar analytical methods.
  • Normative data. As you test your products (and competitive products) over time, the goal is to build normative databases, so that successive product tests become more meaningful and valuable. The normative data, or norms, continually improve a company's ability to correctly interpret product-testing scores, and the norms help reveal exactly how good, or how bad, the test product is.
  • Same research company. Use one research company for all of your product testing. This is the only way you can make sure all tests are conducted in exactly the same way.
  • Real environment test. If the product is used in offices, it should be tested in offices, by people who work in offices. If the product is typically used at home, it should be tested at home. If the product is consumed in restaurants, it should be tested in restaurants; and so on. In general, this kind of "real environment" test will produce the most accurate results. For example, for food products, an in-home usage test is almost always more accurate and predictive than a central-location taste test.
  • Relevant universe. Sampling is a critical variable in product testing. For new products or low-share products, the sample should reflect, or represent, the brand share makeup of the market. For well-established, high-share or highly differentiated products, the sample should contain a readable subsample of that product's users and a readable cell of non-users. If the product category is underdeveloped (e.g., a relatively new category), then the sample should include non-users of the category, as well as users. It's also always important to represent medium-to-heavy users of the product category in the final sample. In summary, if your brand share is very low, it's important to assign more weight, or importance, to the opinions from non-users of your brand. If your brand share is very high, then what your users think is more important.
  • Critical variables. Product performance and quality must be defined from the consumer's perspective, not the manufacturer's. What aspects of the product are truly important to consumers? What critical variables determine the consumer's satisfaction with the product? These critical variables must be identified for each product category (typically, with focus groups or depth interviews) to design an accurate product testing system.
  • Conservative actions. The formulation of an established product should never be changed without careful testing and evaluation of the new formulation. Then, once you are sure you have a better product, introduce it into a limited geographic area for a reasonable period (several product repeat-purchase cycles). Then and only then, roll the new product out to all markets. The smaller your market share, the greater the risks that can accompany a new formulation. The larger your market share, the more conservative you should be in introducing a new formulation.

The major techniques

The most widely used research designs for product testing are the monadic, sequential monadic, paired-comparison and protomonadic.

Monadic testing usually is the best method. Testing a product by itself offers many advantages. Interaction between products, which occurs in paired-comparison tests, is eliminated. The monadic test simulates real life - we usually use products one at a time. By focusing the respondent's attention on one product, the monadic test provides the most accurate and actionable diagnostic information. The monadic design also permits use of normative data and development of norms and action standards.

Virtually all products can be tested monadically, whereas many cannot be accurately tested in paired-comparison designs. For example, a product with a very strong flavor (hot peppers, alcohol, etc.) may deaden or inhibit the taste buds, so the respondent cannot really taste the second product.

Sequential monadic designs are often used to reduce costs. In this design, each respondent evaluates two products. The respondent uses one product, evaluates it, then uses the second product and evaluates it. The sequential monadic design works reasonably well in most instances, and offers some of the same advantages as pure monadic testing.

Be aware, however, of the "suppression effect" in sequential monadic testing. All the test scores will be lower in a sequential monadic design than in a pure monadic test. Therefore the results from sequential monadic tests cannot be compared to monadic test results. Also, as in paired-comparison testing, an "interaction effect" is at work in sequential monadic designs. If one of the two products is exceptionally good, then the other product's test scores are disproportionately lower, and vice versa.

Paired-comparison designs, in which the consumer is asked to use two products and determine which product is better, appeals to our common sense. It's a wonderful design if you're presenting evidence to a jury, because of its "face value" or "face validity."

The paired comparison can be a very sensitive testing technique (i.e., it can measure very small differences) between two products. Also the paired-comparison test is often less expensive than other methods, because sample sizes can be smaller in some instances.

Paired comparison testing, however, has limited value in a serious, ongoing product testing program The paired-comparison test does not tell us when both products are bad. The paired-comparison test does not lend itself to the use of normative data. The paired-comparison test is heavily influenced by the "interaction effect" - that is, any variations in the product we are testing against will create corresponding variance in our product's test scores.

The protomonadic design's definition varies greatly from researcher to researcher. It begins as a monadic test, followed by a paired comparison. Often, sequential monadic tests are also followed by a paired-comparison test. The protomonadic design yields good diagnostic data, and the paired-comparison at the end can be thought of as a safety net - as added insurance that the results are correct. The protomonadic design is typically used in central-location taste testing, not in-home (because of complexity of execution).

Non-packaged good categories

While most product testing is conducted in the food and beverage industries, product testing concepts and methods are applicable to virtually all product categories, although the structure and mechanics of execution will vary greatly from category to category. Computer software, furniture, store environments, dog food , airline service, equipment prototypes - any and all of these products can be tested.

Competitive advantage

The ultimate benefit of product testing is competitive advantage. Product superiority is the surest way to dominate a product category or industry. Companies dedicated to ongoing product improvement and testing can achieve superiority and a competitive advantage of great strategic significance.

Companies that ignore product improvement and product testing, on the other hand, may wake up one morning to find themselves on the brink of extinction from a competitor who has built a better mousetrap.