Examining the potential for response differences based on traditional audience demographics

Editor’s note: Patrick Stokes is founder and CEO of Rep Data, a data collection solutions firm.

There are many factors that weigh into achieving data quality in market research, an issue that is persistent, evolving and complex. One of the best ways to tackle this challenge is by building up our knowledge about all the intricate challenges surrounding quality, so we can improve quality with things like advanced technology, better sourcing practices, expert intervention and more. If researchers truly want to trust data, we must understand every element throughout the research process that is impacting the quality of our insights. 

Our latest research-on-research, conducted in partnership with DM2: Digital Marketing and Measurement, explored how audience demographics, characteristics and behaviors affect outcomes. We asked one overarching question: “How are today’s survey respondents affecting your data quality?” The project examines the potential for response differences based on traditional audience demographics such as age and gender; the type of device used to answer the survey questions; self-reported behaviors and personality types; and other key indicators.

To measure quality, we used a methodology that leverages trackable, quality-oriented question sets used for many years to determine sample provider and respondent quality and characteristics. The longevity of these question sets provided data that gave significant benchmarks for the United States, from 50K+ interviews in the past year alone. In addition, some standard questions from sources such as the U.S. Census were included to give a foundation for outside comparisons. In all, we collected n=1,800 responses on three different survey lengths (7, 12 and 18 minutes), with completes divided evenly by survey length and device type (mobile vs. tablet/laptop/desktop), as well as reflecting consistent age and gender quotas.

Three audience characteristics that impact data quality  

What we found was that there are very specific characteristics among audience members that seem to have a direct impact on data quality. Just a few of the findings include:

  1. Mobile survey-taking continues to impact quality, as those respondents using mobile devices were nearly one-third more likely to provide lower quality data than desktop computer, laptop computer or tablet users.
  2. Education level produced a bell-shaped curve, with the least educated (some high school – 19%) and most educated (master’s degree – 8%) individuals providing data that was of lowest quality, including answer inconsistencies.
  3. Political leanings produced the same bell-shaped curve. Those who identified as extremely liberal or extremely conservative were, on average, 19% less likely to provide quality answers than their more moderate counterparts.

Other profile points such as income levels, self-identified characteristics (such as propensity for risk taking), social behaviors, technology and online usage, and more were also examined in the research. 

Education level produced a bell-shaped curve, with the least educated (some high school – 19%) and most educated (master’s degree – 8%) individuals providing data that was of lowest quality, including answer inconsistencies.

Taking a thoughtful approach to sampling 

So how can brands respond? There are several best practices that can help mitigate impact. From improving respondent experiences across the board, specifically on mobile, all the way to employing layered techniques for fraud mitigation and leaning on expert project managers, the research report includes steps that can be taken to improve outcomes.

Perhaps the most vital piece is to take a more thoughtful approach to sampling. For audiences that fall into high-risk categories, such as those identified by this study, aggregating sample can help to target higher quality respondents by bringing them in from a wider number of sources. This approach can also address the pandemic- and economic-instigated squeeze we are currently experiencing when it comes to demand for respondents exceeding current supply.

We already know that the research ecosystem is changing rapidly and, even with ongoing focus by the industry and its leaders on data quality, there has been little significant progress. From survey design to respondent engagement and attention, to persistent fraud that is becoming more sophisticated, quality can be affected at any stage in the research process. Adapting research practices and building knowledge surrounding data quality challenges is critical to success.