Actions speak louder than words

Editor's note: Megan Copas is consumer research director at 84.51°. She can be reached at megan.copas@84.51.com. Patrick Coyle is chief marketing officer at nutpods. He can be reached at patrick.coyle@nutpods.com.

Research surveys are an indispensable part of making customer-first decisions; however, hard-to-uncover gaps in research quality can lead businesses down faulty paths. The well-worn mantra "garbage in, garbage out" takes on new significance in research methodology – a truth that's often acknowledged but rarely deeply examined. Imagine the cascading consequences of including respondents in a study who have no genuine connection to the brand or category: each false data point is like a small crack in a foundation, potentially causing an entire strategic structure to crumble.

You don't know what you don't know – and in the world of market research, this knowledge gap can be expensive. The unseen biases, the undetected inaccuracies, the subtle distortions that slip past cursory review can lead businesses down entirely wrong paths, resulting in misguided product development, ineffective marketing strategies and missed market opportunities.

To explore the impacts of sample quality on research outcomes, nutpods, a leading maker of dairy-free creamers, and 84.51° – a retail data science, research and media company – conducted an experiment comparing behaviorally verified respondents with self-claimed respondents, yielding critical insights on how a low-quality sample leads to different and flawed business decisions.

The DIY platform challenge 

Nutpods uses survey research to gain consumer insights that inform business decisions. These research insights enable it to develop products and strategies that drive consumer delight. Nutpods uses many different research methods, including do-it-yourself research platforms that empower its team to quickly inform their decisions and hypotheses. However, a close examination of screener and usage data raised concerns about the quality of respondents included in the studies. 

To investigate these issues and find a solution, nutpods engaged 84.51° to conduct a head-to-head comparison of survey platforms – the DIY platform challenge. The study aimed to examine the impact of sample quality on research outcomes, by comparing self-claimed samples with behaviorally verified samples (Figure 1).

By understanding how different sampling methodologies impact business decisions, nutpods could make more informed choices regarding the allocation of its research budget. This allows the brand to determine when to utilize behaviorally verified samples and when a less rigorous approach is sufficient.

Methodology

The DIY platform challenge entailed fielding the same study in two different platforms: 84.51° In-Queries (a behaviorally verified platform) and a well-known traditional DIY platform. The questionnaire was designed to provide insights to the following business questions: 

  • What are consumer habits and practices around plant-based milk, creamer and coffee usage and perceptions?
  • What are the top product attributes that impact purchase intent for plant-based coffee creamers?
  • What are the top claims for the nutpods product and what impact do they have on purchase intention?

Survey respondents were screened for individuals who had consumed plant-based milk and coffee within the past three months. The traditional DIY platform qualified survey respondents based on self-claimed screener questions. In other words, respondents were selected through their own declarations of meeting the survey criteria. 

In contrast, the 84.51° In-Queries respondent group was based on a transaction dataset, including more than 2 billion annual transactions and double-verified their eligibility through traditional screening questions. 

The data collected from these studies was then analyzed, with comparisons made across the responses obtained from the two different platforms. 

Findings

Sample quality and accuracy. Measuring the efficiency of a sample through incidence rate is important because a higher incidence rate means more respondents are likely to qualify for the survey, making the sample-gathering process more efficient and cost-effective. With an incidence rate of 72% versus 42%, the behaviorally verified sample was significantly better at identifying respondents or category purchasers who met the survey criteria (Figure 2). 

Regarding accuracy, the percentage of respondents from the traditional DIY sample who reported purchasing coffee creamers and energy drinks deviated markedly from national averages (Figure 3). This discrepancy raised doubts about the DIY sample due to overstatements in the two categories. 

The “yes to everything” phenomenon. When asked about coffee preparation methods at home, the behaviorally verified sample showed a more realistic distribution of responses, with 6% of respondents selecting every possible preparation method. This contrasted with the traditional DIY sample, where 50% of respondents claimed to use every choice of preparation methods, raising concerns about data quality (Figure 4).

A high percentage of respondents that selected every usage occasion for plant-based milk also made it difficult to identify insights (Figure 5). For example, when asked, “How do you use plant-based milk in your home?” responses from the traditional DIY sample raised concerns about the lack of discrimination between uses, with responses at or close to 70% for nearly every usage occasion (Figure 6).

The behaviorally verified sample revealed more nuanced data: 73% used it on cereal and 41% drank it straight, clearly indicating the more common usage and highlighting potential opportunities.

Impact of claims on purchase interest

One of the objectives of the study was to identify the most impactful product attributes to promote. 84.51° In-Queries identified the claim that most effectively drove purchase interest among respondents (see Figure 7). This claim resonated with 58% of respondents, significantly higher than other claims tested. In contrast, the traditional DIY sample showed a flatter distribution of responses, making it difficult to discern the most impactful claims. 

The study also examined changes in purchase intent after exposing respondents to positive claims about nutpods. Respondents were shown claims such as, “nutpods has zero sugar and only 10 calories per serving,” “nutpods is smooth and creamy without sweetness” and “nutpods has more quality certifications than any creamer or milk brand” (Figure 8). 

The behaviorally verified sample showed a 1.5x increase in purchase intent post-exposure, while the traditional DIY sample indicated that customers were twice as likely not to purchase a nutpods product post-exposure (Figure 9). The finding that purchase likelihood decreased when customers were presented with positive claims further undermined confidence in making business decisions based on the traditional DIY results.

The difference in results between the two samples demonstrated the advantages of behaviorally verified sampling, ultimately reducing the risk of costly errors compared to traditional DIY samples. Behaviorally verified sampling results showed:

Enhanced accuracy and efficiency. The behaviorally verified platform and sample were found to be more efficient and accurate in identifying qualified survey respondents. This method showed a higher incidence rate of category purchasers and provided clearer, more reliable results. In contrast, the traditional DIY platform and sample, which relied on self-claimed screener questions, often resulted in inaccurate data and a higher likelihood of respondents selecting all options to qualify, leading to unclear findings.

Results ensuring reliable and informed business decisions. With behavioral data, marketers can ensure that 100% of the individuals in their study are real people (as opposed to bots) who are genuinely engaged with the category or their brand. This approach enhances the trustworthiness of the results and significantly reduces the risk of making ill-informed business decisions.

Make informed choices 

While the behaviorally verified platform and sampling were more effective in ensuring data accuracy and reliability, this doesn't mean that traditional DIY platforms have no place in market research. Rather, understanding these differences in sample quality empowers brands to make informed choices about their research methodology and budget allocation. 

When precise, verified consumer behavior is crucial for high-stakes decisions, behaviorally verified sampling can offer rigor and confidence. For other research needs where absolute precision is less critical, traditional DIY platforms that use self-claimed sampling may suffice. The key is recognizing when each approach is most appropriate, enabling companies to optimize their research investment while ensuring the quality of insights matches their business objectives.

The diversity of research methodologies reflects the complex nature of consumer insights. Different research approaches offer advantages that extend beyond data collection. Behavioral verification represents a sophisticated lens through which brands can understand consumer motivations, revealing patterns that self-reported data might obscure. In this case, the depth of insight matters more than the breadth of surface-level information. By understanding the subtle distinctions between sampling techniques, brands can develop more nuanced strategies that capture the intricate dynamics of consumer behavior.

As the retail landscape continues to evolve, savvy approaches to research will be crucial for brands seeking to stay ahead of the curve. Researchers must evaluate their research objectives, timeline and budget constraints when choosing between methodologies. Doing so will allow researchers to maximize the value of their investments and obtain the quality of insights needed for each business decision.