By JD Deitch, Chief Research Officer >> P2Sample
What can we do as an industry to improve participation and reduce bias? We have more opportunity now in the digital age to drastically change our research practices to overcome challenges in securing representative research participants. New technologies, methodologies and approaches are now allowing us to improve the respondent experience and give them respect, prioritize reach and diversity and put quality front and center.
It’s no secret that we are some distance away from accomplishing these goals. In fact, many are still scrambling to execute the basics that are being demanded by changes in the social, economic and digital landscape. Innately connected consumers – the so-called “digital native” generation – aren’t joining traditional panels. Individuals are demanding more from every interaction, online and otherwise. Transparency and accountability are permeating every aspect of life. Those who don’t get it right in the market research space will find themselves increasingly sidelined.
Rapid transformation
When we examine the world of sample specifically, i.e., how participants are sourced, how much it costs to access real respondents and the future of panels themselves, there are several things that should be occurring in the midst of rapid transformation. Some of the challenges that remain include:
- Poorly designed surveys. We live in a world where great attention is spent on user interfaces and processes that reduce friction, where information is easily accessible and even complex transactions are made simple. And still there are tons of poorly designed surveys and surveys that don’t properly render regardless of screen size. There are still high-friction interactions for panel registration or during the survey experience as users are bounced from router to router (being asked their age and gender every time). The whole thing sticks out like a sore thumb.
- Dismal interactions. Even those respondents who like to share their opinions don’t see panels or surveys as the first place to do so. They can make their voices heard everywhere from Yelp! to Amazon to Google reviews to Reddit. Research methodologies have evolved to include engaging voice and free-form chat as methods for data collection. Respondents can interact directly with brands, companies and their peers to speak their minds. And they often get immediate gratification in the form of a response, aggregate or otherwise. Are people going to stop needing surveys or taking surveys? It’s very possible.
- Payoffs that suck. Respondents figure out fairly quickly that they have to take a large amount of surveys to earn enough points for any kind of payoff. Incentives are small, hard to earn and don’t resonate with the survey respondent. Some clients are recognizing that this has an impact on participation. Some aren’t.
In addition to these enduring market research issues, many researchers largely don’t understand sample supply and many continue to operate under assumptions that haven’t been true for 10 years. Just some of these misunderstandings include a lack of comprehension of how real-time sample works in the age of automation, a misconception that double opt-in panels are “better” and the misguided belief that fraud is easily solved with traditional, static measures. In fact, sample is a dynamic space that, in some cases, is creating solutions through nimble use of technologies like artificial intelligence and automation. Those that aren’t using technology to solve problems are being left behind.
Moving the dial for solid data outcomes
There are several key areas where we can start to institute real change and some companies are already making strides toward this future. It’s time to examine how the digital world has opened up new avenues for uncovering quality data.
Reach and diversity
Aggregating respondents from multiple sources is a practical solution for creating diversity. By pulling in individuals from a wide range of outlets such as affiliate networks, publisher networks, shopping sites, social media platforms, ad agencies, blog networks and loyalty sites, there is a better chance of obtaining a diverse and representative cross-section of the population. As suggested above, non-panel sources are more important than ever in the face of declining participation.
Automation allows us to optimize this process and benefit all stakeholders. Some sample providers are making investments in programmatic acquisition, which in turn allows recruitment at scale that evolves alongside the market, providing a competitive advantage. When this manner of recruitment is done right, providers can efficiently source from hundreds of sample suppliers of all types, thereby providing diversity while also minimizing other biases that arise with concentrations of certain types of people.
This kind of shift naturally leads to the need for a new way to measure quality. The traditional static look at counts of registered participants by demographics and response rates is effectively meaningless for judging a supplier’s ability and dependability to supply respondents for a given project. What matters now is real-time feasibility across an ever-evolving collection of sources and the headroom across those sources to facilitate growth. Reach across numerous diverse sources is the only way to increase feasibility and reduce coverage bias.
Respondents will be rewarded, respected and heard
The entire model for incentivizing respondents is beginning to change, as they are no longer content with earning points and getting small rewards. Many suppliers have now figured out how to improve incentive choice and have streamlined the fulfillment process to meet consumer demands for choice and instant rewards. This is actually the easy part.
Automation and artificial intelligence allow us to improve respondent engagement from multiple angles. While the incentive definitely plays a role, and customization and personalization contribute to its overall perceived value, engagement has more to do with making things better for respondents all around. As researchers, none of us should underestimate the impact of engaged and happy respondents on data quality – and future participation!
Automation and artificial intelligence give sample companies the tools needed to improve the respondent experience from the ground up. Just a few of the things they can do for us:
- quickly identify hidden quotas to present the study only to the appropriate, qualifying audience, even when correct parameters were not provided up front;
- detect users who look like they are become disengaged or are ready to stop participating and try to retain them by directing them to highly rated, easy-to-complete surveys; and
- analyze the vast pool of user data to determine habits, like completion rates overlaid with times of the week, in order to optimize delivery of appropriate surveys.
With problems like router bouncing and more that create negative user experiences, directly managing the process in field to stop bad experiences is a must. This can include everything from employing automation to spot trouble in field – trouble which can be algorithmically identified by using things like low complete rates and even user-generated survey ratings.
Engagement can also improve when suppliers take advantage of the millions of data points that are available to them. Using this data, backed by automated algorithms, surveys can be targeted appropriately so users spend less time trying to qualify for a study and find a faster path to a good experience.
Automation also allows for a complete feedback loop, in which respondents can be rated based on their behaviors (such as thoughtful responses versus speedlining) and respondents can also rate the surveys they take. Their feedback can be coupled with algorithms that automatically monitor dozens of field statistics to gauge the respondent experience. At P2Sample, we promote great experiences through increased sample flow, while bad ones have their traffic slowed and may even be quarantined. Bad experiences cause participants to disengage or drop out, which leads to bad data. This kind of forward-thinking approach can help boost engagement and satisfaction, while also eliminating avoidable delays and weird data that only become apparent once the project is complete.
Modern fraud mitigation
Online research, due to its relatively high payout, has attracted its share of fraud. Estimates of fraud in surveys range from 5-20% of completes, sometimes higher. In the early days of the internet and online research, fraud was somewhat confined due to lack of automation and other technologies that now make it easier to trick the system. The methods and barriers we used in these early days to mitigate fraud will no longer work in the age of automated digital recruitment where speed, scale and sophistication benefit both good actors and bad actors.
The bad actors can now use technology to launch thousands upon thousands of attacks on panel company servers and even include the trading of stolen credentials in the dark web. These are not “bots,” as so many in the industry call them, as if they were amateurishly written programs. These are humans working with machines and they are easily defeating the industry’s “best” tools. Conventional techniques (think e-mail validation and Captcha) must now be combined with more advanced techniques to be effective.
One advanced method that works is the smart use of machine learning, a subset of artificial intelligence that significantly enhances detection. Using billions of data points and leveraging domain knowledge at the outset, machines can be trained to recognize unusual emerging patterns like surges in identical user profiles. This kind of system intelligently adapts to changing threats.
Equally important to containing fraud is the capture of personally-identifiable information on the individual. While this comes with high standards for data protection and privacy, it greatly facilitates fingerprinting and fraud detection. It also eliminates one of the main concerns about real-time sample, as it results in a concrete opt-in event and permits all kinds of work that was heretofore impossible, like recontacts and product tests and even offline qualitative work.
Further considerations
The above is just scraping the surface of the changes that are occurring in the digital landscape for our industry. For years, we have generally focused on the tactical benefits of technology, such as using automation to reduce costs and uncover insights more quickly. But the “north star” for us as an industry has always been, and must continue to be, providing data that our clients can trust.
The companies that are committed to this outcome are set apart because of the way they use the technologies that are now available to them. They are using things like automation and artificial intelligence not just to meet demands for speed and profitability but also to deliver materially better, more accurate data. A focus on things like improving the respondent experience, reducing fraud, improving targeting and proactively solving problems in field begins to create a virtuous circle that improves outcomes for everyone.