A threat not a crisis
Editor's note: Mike Booth is director of data and analytics at InspiredHealth.
Getting good-quality data from research panels has been a challenge ever since research panels became a thing. But let’s be honest – it wasn’t always a major challenge for the research industry. Data-checking techniques meant we could all stay one step ahead of bad-faith respondents – those only in it for the incentives. But in recent years there have been increasing concerns about respondent quality and for good reason.
What’s changed?
- There are more online survey panel providers than ever. Many panels are linked through aggregators and subcontracting, making it unclear who is responsible for quality assurance and whether the same level of quality is guaranteed throughout field.
- A rapid increase in the availability of consumer privacy tools is making cheating easier and more accessible than ever.
- Easily available automated survey-taking software has become prevalent. For instance, Browser Automation Studio allows a user without any programming knowledge to create a bot and deploy it repeatedly on a survey.
- The gig economy created professional survey takers who are in a rush to cash in and may not be reliable or honest.
- There are highly motivated fraudsters in foreign, non-target markets who stand to make a substantial amount of money relative to their native economy. For instance, a fraudster acting from Bangladesh can make $1,500 per month in survey incentives, or $18,000 per year, which is almost seven times greater than the average income of their country.
While there is growing chatter in the corridors of research conferences that, at some point in the future, the bots could take over or sample quality issues could render the panels useless, the reality is that panel companies are stepping up, innovators are stepping in and as industry we are pushing back against sampling issues to maintain our integrity. But there is a problem for agencies to solve: Not all panels provide the same level of security and quality. How can a market researcher looking to contract a panel know what type of quality they will receive? And if the panel contracts with other vendors, will the quality be the same and is there any guarantee?
Hard to do your job
Why should you, as a researcher, care about data quality and go to the lengths required to verify that you are receiving high-quality respondents? Like, really? When data is bad it’s hard for you to do your job. Our job is to analyze data, identify interesting relationships and extract inspiring insights and use them to aid decision-making. But how do you know that the survey responses themselves – the bedrock of your analysis – are fit for purpose?
Consider for a moment the different types of survey responses you can get using the symbolic classification system shown in Figure 1.
So, bored? Fraud? Which is it? Bored respondents are our fault. Overly long surveys, poorly worded questions, complex answer grids – it’s enough to make anyone tune out for a question or two. Fraudulent actors, on the other hand, are a different problem and can be difficult to distinguish from bored respondents unless you look for certain clues.
Fraudsters are real people acting in bad faith and can comprise 5-25% of respondents. They pretend to know the product category, lying their way through screeners. Once they know how to get past the screener, they can sell that information online, post it on web forums and repeatedly complete your survey for the incentives. There are Reddit communities built around doing exactly this and profiting from it, with people posting their monthly earnings of survey income as a badge of honor and bragging rights. And now the frauds are leveraging AI to infect surveys with greater efficiency than ever before.
Lately our team have come across a new crop of open-ends that we call “beauty queens,” created by ChatGPT or other AI language tools. They are perfectly worded responses, expertly crafted and sound too good to be true. Because they aren’t true!
Peeling back the layers
Quality-verification methods can be divided into roughly four categories: device- and browser-specific signals; survey-specific behavior; content of survey responses; and the broader survey-taking environment (something panel providers have access to but most agencies do not).
There are many device-fingerprinting signals one can gather about a respondent. To be clear, these signals do not on their own indicate fraud but they are often used by frauds. Several are consumer-level privacy tools that are used by honest people and should not on their own disqualify someone. But by tracking each and looking at combinations of their use, you have clues to add to survey behavior and content to get a clear view of a respondent.
VPNs: Virtual private networks mask the user's real IP address, allowing fraudsters to appear as if they are accessing surveys from different locations than they are in and can also encrypt internet traffic.
Incognito mode: A browser feature preventing the storage of browsing history and cookies, aiding frauds in avoiding detection when reentering surveys.
Virtual machines: Allow a user to run multiple instances of an operating system, enabling frauds to access a survey repeatedly or multiple surveys at the same time, while appearing to be different people/devices.
Device tampering: Alters device settings or configurations and can help bypass security measures of surveys.
Privacy settings: Adjusting these settings can limit the amount of detectable information about a user’s device and browser.
Tor browser: A browser that largely anonymizes internet activity.
Android emulators: These simulate Android devices on a computer, allowing frauds to mimic being different users for survey entry.
Jailbroken devices: These have removed restrictions, giving frauds the ability to modify device properties or install unauthorized apps.
Geolocation spoofing: This involves faking one's geographic location, enabling frauds to access location-specific surveys that are meant to prevent users outside that region.
To peel back the layers of what a respondent is doing in the survey, the old tools are no longer enough. Frauds have learned the game. More innovative tools are needed to capture things you can’t see in the survey data alone.
The following are tried-and-true tools for flagging bored respondents that are no longer useful for catching fraudulent respondents:
Speeding: Your fastest five to 10 percent of respondents are largely not paying attention and not comprehending the survey contents. But those are likely your bored respondents. Frauds know better and have learned to program bots to move at a reasonable pace or to themselves move slowly while taking multiple surveys to increase profits.
Straightlining: Giving the same answer repeatedly on Likert scales will show that a respondent is bored but frauds know this will get them kicked out or flagged and are unlikely to do it.
Attention traps: The “click Answer 2 if you are paying attention” trap is not going to trip up the professional survey takers or the frauds who have seen this trick hundreds of times. If you are going to use traps, they should aim to catch someone answering dishonestly, which is what frauds do. Either through testing their content knowledge or asking questions that set up potentially contradictory answers, you need to monitor for dishonesty.
Open-ends: There is a lot of innovation happening in this space with AI and text-based machine learning models that make classifying the quality of open-ends fast. But at the same time AI poses a new problem. The open-ends are no longer silver bullets for detecting fraud: the beauty queens can now sound product-educated. We’ve even seen some beauty queens try to disguise themselves by muddying themselves up a bit with intentional spelling and grammatical errors.
Some newer methods that can detect fraud include:
Detect copy-and-paste and odd behavior in open-ends: The beauty queens need to be dealt with and detected. To remedy the situation mentioned above that open-ended content can no longer reliably expose fraud, more sophisticated analyses of behavior are needed. The key here is that behavior is now the thing to watch: like monitoring characters typed per minute, the use of copy-and-paste and looking for repeat or questionably similar open-ends across respondents. We’ve seen some AI-generated answers across respondents in the same survey that are nearly identical, enough to then look further into device-specific metadata and determine they were generated by the same person.
Atypical mouse movements, page navigation and browser rendering: Bots try to hide what they are. Many third-party security plug-ins can detect programmatically driven browsers like headless Chrome, but some bots are subtly obfuscated, so the more sophisticated the detection of anomalous behavior, the better.
Developer console open: No doctor or regular consumer should have the developer console open while taking a survey. Altering the html code of a webpage is how frauds block scripts and other plugins that would otherwise catch them.
Random choosing: As mentioned, watching for straightlining is not enough to catch frauds, but in its place you can use statistical tools to differentiate and classify atypical response patterns from the larger sample. For instance, in a conjoint study, isolating and removing random choosers. In a large grid, using dimension reduction techniques or latent-class analysis to isolate anomalous patterns.
More tools
The panel providers have more fraud-detection tools at their disposal. Panels can see what is happening across multiple surveys from different agencies, as well as the response history of any single respondent. As mentioned at the outset, the proliferation of panels and the sub-vending that occurs leaves agencies uncertain as to whether all panels involved in a recruit are doing these things.
In our opinion panels can and should:
Identify professional survey-takers based on survey response history. This requires the same respondents to have the same ID or a means of linking IDs across panels.
Flag problematic respondents. If a respondent has been flagged as low-quality by multiple agencies, it’s time to remove them from the panel. This requires agencies and panels working together to submit quality deletion knowledge and reasons and the panels to aggregate that knowledge.
Pre-screen respondents: Use many of the fingerprinting and survey-specific techniques discussed above during panel screen-in surveys.
De-duplication for cross-panel repeats: The more panels involved in a single sample recruit, the higher the duplication rate. Panel providers can and should cooperate to prevent duplicate survey entry when subcontracting.
Registration security: IP blacklists, Captcha systems and filtering out registrations from temporary and throwaway e-mail addresses.
Validating professional credentials: Require licensure proof for professionals and cross-checking with publicly available data.
Many panels do these things or a combination of them. However, in our experience panel quality varies substantially. As mentioned, the sub-vending across panels can create issues; for instance, a lower-quality panel is brought on at the end of field to get the last bit of hard-to-reach sample. If you are not watching your data closely, you will not know if these sub-vendors are providing the same quality as you got at the outset. Panels could help here by being transparent about sub-vending and tracking quality of their sub-vendors to remove or remediate unreliable vendors.
Front-door security as prevention
Looking at the above tools discussed, it’s clear that agencies and panels should be working together to prevent fraud: each has their own opportunities and strengths to combat the issue. In our experience, there is one more thing an agency can do to protect against fraud and it’s an important one. The front door to your survey is the screening section. The more times a dishonest actor accesses your survey, the higher the probability they figure out your screening criteria and let themselves in. Stopping repeated survey touches should be one of your highest priorities.
I advise against using an open survey link – one that is the same for all respondents. They are easily shared, can be used repeatedly by the same person and even if you add a unique query string label it can be modified to break the protections you’ve added. Use restricted survey links that are unique for each respondent with a complex hash key and only allow those links to open the survey. For instance, our survey URL links contain a unique 108-alpha-numeric-character string for each respondent. Only links generated by us can access the survey, meaning the probability of reentering our surveys by guessing the hash of another of our links is about 4.69×10−195 – which is essentially zero. Each link can only be used once, then it’s dead.
By combining unique, restricted survey links with advanced fingerprinting methods, you can block people visiting the survey repeatedly from the same device. That severely limits a fraud’s ability to take the surveys multiple times and crack the screener. For example, if you signed up with a panel using six throwaway e-mail addresses and have six unique links to our survey, after you use one link and screen out, if you try another link from the same device, we kick you out. Same goes if you complete the survey. You will only have as many attempts as you have combinations of links and unique devices that we haven’t blocked. We are making it a pain in the neck for someone to guess their way through our screeners or take the survey multiple times!
The solution is collaborative
You may be reading this and thinking, “The panel we use says they are constantly looking into this and they say they have the highest-quality respondents.” Many panels employ security but not all security systems are equal. We’ve seen failures in security from providers touting rigorous protocols dozens of times. I don’t necessarily blame the panels; the incentives are misaligned. Panels want to make money providing as many respondents as possible to agencies, while we as agencies don’t want to pay for low-quality respondents and require strict timelines for field. Some audiences are hard to find and frauds commit fraud, which means bypassing security. It’s a complicated issue and the solution is collaborative.
I mentioned at the beginning that I don’t believe as an industry we are in a respondent quality crisis. I do, however, believe it’s a threat we must continue to respond to. Based on our company’s experience doing extensive verification of panel quality from most of the major vendors, we believe in doing due diligence to verify data quality in an ongoing, systematic manner.