Norm Leferman is president of Leferman Associates, a Southborough, Mass., research firm.

155028494Bruce Isaacson’s article on claim substantiation studies (“Playing nice with legal”) in the January 2013 issue of Quirk’s provides several very good recommendations for how to best substantiate a product claim. In my opinion, though, he overlooked what could be a potentially damaging consequence of testing multiple claims in a single study.

While Isaacson correctly states that “When substantiating claims, questionnaires must focus only on the claims of interest and nothing more,” he also states that “A claim substantiation questionnaire may test a number of different claims and marketers can use the results that support their product, even if other results do not.” That is a very dangerous tactic, particularly when you are trying to make a comparative claim against a named brand.

Here’s why. Suppose you set out to substantiate that your cake mix (continuing to use Isaacson’s category example) is better than Duncan Hines. In the context of a properly constructed product test you not only question consumers as to their overall ratings of each product/brand but also ask them to rate each product on flavor strength, moistness and each of several other key attributes/benefits.

Let’s suppose for the sake of argument that the client’s product is rated as highly as Duncan Hines overall (e.g., there is no claim of overall superiority) but wins significantly on flavor strength. And, just for the sake of argument, Duncan Hines wins significantly on moistness.

Many marketers, based on this research, might be encouraged to create a communications campaign that touts their cake mix as being more flavorful than Duncan Hines. Why not? The data supports it. And, legal and the media would accept this research as adequate substantiation.

Here’s the problem. Duncan Hines, without much difficulty, can ask to see the substantiation, including the questionnaire. Once the questionnaire has been revealed, aggressive attorneys at Duncan Hines can see that multiple issues were studied and can get hold of the entire data set. Guess what? Your own study that substantiates your claim of more flavor now can become the substantiation for Duncan Hines to make a counter claim of “more moistness.”

Isaacson also suggests in his article that failure to win on the intended claim in a study might still reveal another possible claim: “We can state claims that closely match the data even when the tested product did not ‘win’ the test in the traditional sense of the word.” This can also be risky. Suppose, using Isaacson’s example, that you launch a claim that one-third of consumers prefer your product. When the aggressive lawyers for another brand obtain that study, they may find that their brand actually did even better. You get to keep your substantiated claim but they use your own data to substantiate an even better claim.

The safest way

So what should you do? Even though it is more expensive, the safest way to substantiate a claim is to conduct two studies. First, conduct your rigorous and carefully-controlled product test to determine your potential claims. Then, based on that learning, conduct a follow-up study that only asks the single question that will provide the basis for your claim.

It may cost you more for conducting a second study but it eliminates the potential risk of providing your competitors with ammunition for a counterclaim.