Avoiding common pitfalls in concept testing

Editor’s note: Nitin Sharma is CEO of Gold Research, San Antonio, Texas. This is an edited version of a post that originally appeared here under the title, “Common pitfalls in concept testing and how to avoid them.”

Concept testing is often a key step in new product development. If not done right, it can produce useless or (even worse) misleading results. Here are the most common pitfalls associated with this type of research and how to avoid them.

1.      

Problem: Researchers often test ideas which are not fully specified. When this happens, respondents have trouble answering purchase intent or other overall interest questions. Alternatively, respondents may interpret what’s meant to be the same concept in many different ways, so their answers aren’t comparable.

Solution: Think like a respondent – what would they need to know about the concept in order to make a simulated purchase decision? If you (or your client) can’t provide this information, then the concept may not be ready for testing.

Problem: A special case of inadequate specification is missing or bad pricing. Asking purchase intent when price is unknown isn’t really meaningful, while asking purchase intent when price is unrealistically high or low yields distorted results.

Solution: Conduct some form of pricing research if you (or your client) can’t provide reasonable prices. Alternatively, use the concept test for other purposes (prioritizing a list of options, assessing characteristics or perceptions, etc.).

Problem: Most new products or services are intended for a certain target population. If your sample is too broadly defined, your results will be distorted by the inclusion of opinions from people who you’re not interested in. If your sample is too narrowly defined, your results will be distorted by the exclusion of opinions from people who you are (or should be) interested in.

Solution: Take the time up front to define your target audience in terms of demographics, category behavior or other characteristics. This definition can then be implemented through sample specifications and/or screening questions. Quotas or weighting may also be needed.

Problem: The main objective of most concept tests is to assess purchase intent. To save money and minimize respondent burden, many researchers stop there. However, this information is of limited value if you don’t know why intent is high or low. In fact, understanding a concept’s strengths and weaknesses is sometimes more useful than estimating overall interest.

Solution: Include questions about concept characteristics. These may be broad or specific, and may be functional or emotional. Then use an appropriate analytical technique to determine the relative importance (impact on overall interest) of these characteristics.

Problem: Many researchers select their data collection method for concept tests based solely on cost and timing considerations, which usually favor the Internet. However, this may not be the right choice. For example, graphics may not show correctly or effectively, especially on mobile devices. In addition, some target audiences (very young, very old, low income or limited education, etc.) may be hard to reach online or are uncomfortable with online surveys.

Solution: Try to choose the methodology which yields the most valid and reliable results, even if it isn’t the fastest or cheapest. Saving time and/or money doesn’t do you any good if it comes at the expense of data quality. Consider using mixed modes or having paper backups if collecting data in person.

Problem: Cost considerations force many researchers to go with the smallest sample size that will provide a reasonable level of precision at the aggregate level. However, small samples often won’t support potentially important breakouts by key respondent characteristics. You may also want to limit some questions to certain sub-samples, such as people with a minimum level of interest in the concept, and here again small base sizes can be problematic.

Solution: Focus on the smallest sub-sample for which you want to be able to make statistically meaningful inferences, and let that drive your total sample size. You can sometimes justify the cost of a larger sample by adding questions which address other related business issues or alternatively by including your concept test as part of another survey.