Gaining the right marketing research insights

Editor’s note: Stephan Basson is the content marketing manager at Factworks. This is an edited version of an article that originally appeared under the title “How Cognitive Testing Improves Market Research.”

In the fast-paced industry of market research, the pressure to accelerate processes is ever-present. One may contemplate bypassing certain steps to expedite data collection. However, in doing so, you risk compromising the integrity of your insights. For instance, what if down the line you discover that respondents misinterpreted your survey questions? This will create skewed data and could potentially lead to decision-making that’s not based on actual customer insight. 

To illustrate, imagine you are a coffee bean producer and you are sending out a survey to learn more about people’s coffee consumption. In your survey, you ask the question “How many cups of coffee do you drink per day?” Now this question seems simple enough to answer, however, it introduces ambiguity in at least two aspects: the definition of a "cup" and the type of "coffee."

The term "cup" can be interpreted differently; some may think of a standard six-ounce cup, while others may consider a larger or smaller serving. Additionally, "coffee" can include various types such as brewed coffee, espresso or specialty drinks, each having different caffeine content and serving sizes. As a result, respondents may provide varied and potentially inaccurate data due to the lack of specificity in defining the unit and the type of beverage. Ultimately, the information you will get from this question tells you very little about how much ground coffee people consume. 

So, how do you go about bringing clarity to this ambiguity? Conduct a cognitive test with your survey. This indispensable tool in the market researcher’s toolkit acts as a safeguard against misunderstanding and ensures the delivery of superior data quality.

How to conduct cognitive testing

To facilitate cognitive testing, an interviewer can administer the survey questions to respondents and encourage them to share the thought processes they go through when answering specific questions. Alternatively, the interviewer can observe the respondents clicking through the survey themselves and ask probing questions. There are many techniques that can be used.

Thinking out loud

The interviewer asks respondents to provide free-form, stream-of-consciousness style answers about what is going through their minds as they answer the questions. For example, an interviewer can ask:

  • Walk me through your thought processes as you answer this question.
  • As you answer, tell me what’s going through your mind.

This method is especially helpful if you want insight into how respondents recall information, which can potentially reveal related concepts worth exploring further. 

Active probing

The interviewer asks planned (proactive) or spontaneous (reactive) questions to delve into specific points or follow up on respondents’ remarks. The former approach poses prewritten questions to all respondents to explore a certain research question, for example:  

  • What does that word or phrase mean to you in the context of this question?
  • How did you come up with your answer?
  • In your own terms, can you explain what this question is trying to ask?

Reactive probing occurs when the interviewer wants to further explore respondents’ unpredictable answers to reveal insights the script doesn’t account for, for example:

  • Why did you mention X (unexpected term)? 
  • Can you tell me more about your answer? 

Interviewing a variety of people helps uncover a wide range of perspectives and potential pitfalls. Research shows that as your sample size increases, additional problems continue to be detected (Blair and Conrad, 2011). As a rule of thumb, we recommend conducting at least 10-15 interviews or more if the survey is very complex and research constraints allow. 

For multinational research, it can also be helpful to recruit participants from various markets to uncover regional differences in how the subject matter is understood. For example, in a past study we presented property rental types in a list to different countries and through cognitive testing we discovered that respondents from Brazil didn’t recognize condos or condominiums as a common rental term. Backed with this insight, we could amend the survey and hide this item from the list of options for Brazilian respondents.

What are the benefits of cognitive testing? 

One can think of cognitive testing as a sort of “insurance” for market research. By that, we mean that it helps to ensure that the data we collect reflects how respondents really think and feel.

By conducting cognitive testing we can:

  • Ensure that respondents understand the question as it was intended.
  • Confirm that respondents can recall the appropriate information required to answer questions.
  • Ensure that respondents can find their answers in the provided item lists or answer categories.
  • Discover unexpected pitfalls in areas that were not considered problematic before.

For clients, this means you gain an improved survey design that works as intended and captures high-quality data. As a bonus, it also allows you to ask some qualitative interview questions which can add more texture to your quantitative data. 

Cognitive testing is a crucial step in market research and its importance should never be underestimated. To make informed, data-driven decisions, you should ensure that your data must first and foremost be accurate and truly reflect the feelings and thoughts of your customer base. 

Sources

‍Blair, J. and Conrad, F.G. (2011) ‘Sample size for cognitive interview pretesting’, Public Opinion Quarterly, 75(4), pp. 636–658. doi:10.1093/poq/nfr035.