Navigating the fraud funnel
Julia Isaacs is junior strategist at KNow Research. She can be reached at
ji@knowresearch.com. Shira Glickman is senior strategist at KNow Research. She can be reached at sg@knowresearch.com.
In the summer of 2022, KNow Research, a qualitatively focused research agency, embarked on an internal project to better understand the widespread issue of participant fraud within the insights industry. The motivation for this investigation arose from a significant 19% uptick in the presence of bad actors infiltrating our virtual fieldwork. Recognizing that this issue extended beyond just our organization, we started an exploration to gain a broader understanding of the issue and ensure that the insights we gather truly stem from genuine human experiences.
Our goal: refine our internal practices and provide thought-starters and resources to the insights industry as a whole to protect the integrity of qualitative insights and avoid scenarios like this:
You recruit 12 participants for one-on-one interviews from a quantitative study that targeted U.S.-based, C-suite professionals. You do your due diligence to invite those who gave thoughtful and relevant responses in the initial quant survey. You schedule follow-up interviews and, to your initial delight, you hit the nail on the head with some outstanding professionals who were exactly who you hoped had taken your survey!
Until you meet “George Henry,” a participant who claims to be a CEO of a New York City-based tech company. The time arrives, you’re on the line and “George” enters the virtual room, though he doesn’t turn on his camera. You greet him and ask that he kindly turn on his webcam. He claims to struggle to figure out how to do this and asks if it’s necessary. You remind him that this was part of the criteria laid out for the study and that he try again. At last, his camera turns on and you see a shadowy figure in a dim room. You ask if he can turn a light on or move into a brighter area, to which he replies that he doesn’t have a brighter room.
You begin the conversation with a typical warmup – what do you do for work and for fun? To which “George” replies with a vague answer that’s difficult to understand. You proceed with some questions to confirm his role in topic of the research, to which he answers cryptically, “Yes, I decide on that, and think are important to employees.” He is soft-spoken and with a thick accent and you worry: He does not appear to be CEO of a New York tech company but how can you know for sure? After a couple minutes of choppy, elusive conversation, you make the call that he is not in fact the person he claims to be and dismiss him, though without any concrete evidence. You feel uncomfortable and awkward in doing so – what do you say? You decide to use the excuse of a bad connection and that you can’t hear him well enough to proceed, thank him for his time and end the call. You later confirm by running his IP address that “George” was located outside the U.S. Your suspicions were right – he was not who you thought he was nor needed for your study. You’re now down one interview, don’t know how to explain the situation to your stakeholders, are concerned about the quality of the overall sample and don’t know how to prevent this from happening again on this study – or any other for that matter.
Defining truth and validity in qualitative research
As qualitative researchers, we are compelled to be lifelong searchers of truth, acting as conduits between the humans who offer seeds of their truth and the stakeholders eager to plant and cultivate them into strategic decisions. The inherent nature of our research positions us perfectly to serve this very purpose, as our immersive methodologies and organic conversations enable us to gain profound insights into the behaviors, motivations and feelings of the diverse audiences we engage with.
As any researcher will tell you, your insights are only as good as your participants. We recognize that the quality and validity of our research depends entirely on the human voices that contribute to it. In qualitative research, researchers often rely on a limited number of participants to gain insights into a larger cohort or population. The ripple of even one person stretching the truth about their identity or experiences can undermine the validity of the entire research endeavor. Therefore, recruiting diverse and knowledgeable participants can be regarded as the most important step in the qualitative research process.
Go straight to the source
In the beginning of 2021, our company saw a large uptick in fraudulent instances, regardless of recruitment method. This prompted us to launch our own initiative to unpack the problem at large, which included going straight to the source – the fraudulent participants themselves. Through our investigation, we discovered that participant quality in qualitative research varies along a spectrum, reflecting the degree to which participants may bend the truth to qualify for a study.
Type 1: Ideal participants. Thoughtful responders, meaningfully contribute to the conversation and honestly represent the target audience. May participate in studies as a part-time job but never as a full-time occupation.
Type 2: Less-ideal participants. May rush through questions and provide bare-minimum responses in the initial screener. This cohort can be considered “professional participants,” as their full-time occupation revolves around actively seeking research studies and finding ways to game the system to qualify for a study (e.g., lying about their age or other demographics).
Type 3: Fraudulent participants. The most severe level of fraud. These individuals intentionally and significantly alter their identities, creating personas that bear no resemblance to their true selves and make up information about the topic at hand.
To learn directly from the less-ideal and fraudulent participants that had been infiltrating our studies, we adopted a two-pronged approach. First, we conducted an in-person focus group with professional participants of the less-ideal type, individuals who had taken part in five or more studies in the past year. Next, we conducted virtual, one-on-one interviews with fraudulent participants, individuals who had attempted to join previous studies but were outed as falsely claiming to be based in the U.S. through IP address scans. We also discovered this cohort uses virtual private networks (VPNs) to mask their true location.
Findings from Type 2 respondents
We conducted an in-person focus group to learn from this cohort. These individuals were open and eloquent, offering extensive, detailed explanations in their responses, and were quite transparent – everything a qualitative researcher could ask for. However, they violated industry recruitment best practices. In participant screeners, along with the standard demographic questions, we build in questions to weed out people who have taken research studies recently and studies on the topic of interest. This group has typically figured out ways around these questions and they often bend the truth to make their way into studies.
Here's what we learned about the world of a professional participant:
- They proactively scan the web for studies in places like social media, Craigslist and Reddit.
- Age is the most common aspect respondents may lie about to participate – they know what the cutoffs tend to be.
- Some told us that they were encouraged to stretch the truth in a screener to qualify for a study.
- For many, qualitative research is their main source of income, drawing an average monthly income of $3,000-$5,000.
Findings from Type 3 respondents
We conducted webcam interviews to learn from this group and carefully recruited participants whom we had previously identified as fraudulent – the ones we had to awkwardly dismiss from interviews and groups. We began like we would any other study – sending out an invite to a list of e-mails we had collected from past studies with a link for them to book a slot to discuss “the research experience” with us for a $50 incentive. We opened up a variety of slots to allow participants flexibility to book but did not anticipate what happened next: within minutes, we had 100+ signups! The invitation link had been passed around like wildfire. Retrospectively we should have anticipated this – of course the link was shared to a broad network; we are recruiting fraudulent participants! We canceled those who were not on our original list, saying that it was only open to those on our original database.
These participants are very good at sticking to their stories or the personas/characters that they create to qualify for a study. Even when presented with their discrepancies in their locations or stories, they refused to budge, insisting they were telling the truth. Here’s what we learned from the fraudulent respondents:
- All had IP address discrepancies and made excuses (i.e., visiting relatives or had traveled to a different coast at the time that they signed up for our study) for the different time zones.
- Many had blacklisted IP addresses or VPNs, which was determined by using a screening tool to detect fraudulent web activity (IPQualityScore).
- Many used a Google Voice number to maintain anonymity.
- Most preferred digital, non-brand-specific gift cards but were not opposed to mailed incentives. We later discovered that some scammers use a U.S. post office box and have a locally based person collect incentives to then mail or deposit to their actual address or account.
We also collaborated with our expert partners, including panel providers and recruiters, who play a crucial role in finding high-quality participants for our studies. Their perspectives helped our team compile a list of proactive and reactive measures to ensure quality participants.
Proactive measures
- Prescreen via phone, being wary of Google Voice users.
- Include and review open-ended responses and trick questions in surveys and screeners.
- Use video audition questions in the screening process.
- Lock survey links so they can’t be shared.
- Use automated tools that screen for habitual survey-taking, inattentive responses and scanning for bot farm networks.
- IP-screen (digital fingerprinting) to examine participants’ past activity, e.g., frequent survey-taking or other acts of fraud.
- Keep fraudulent participants in a database, as opposed to removing them. This provides a log of fraudulent activity and decreases the likelihood of bad actors attempting to join the panel again under a different identity.
Reactive measures
- Go back to your proactive tools if you notice fraud popping up once you’re in field.
- Re-screen recruited participants in the virtual waiting room before a session begins.
- Ask warm-up questions that give you confidence that they are who/where they say they are, e.g., weather-related questions, specific questions about their work and where they are located (while cross-checking their IP location).
- If you still aren’t confident, you can ask then to show their photo ID to verify their identity (use at your own comfort level and discretion).
Through our research, our team developed the concept of the "fraud funnel," which identifies three levels of fraudulent instances: the initial screening, participant scheduling and fieldwork. Organizations should aim to detect and remove bad actors at the earliest stage in the funnel before they reach the coveted fieldwork seats.
Anticipate and prevent
Fraud and scams in research, like in any other industry, can never be completely eradicated. Instead, it is important for organizations to proactively anticipate and prevent them to the best extent possible. The focus should shift from asking if fraud will occur in a study to understanding to what extent it may happen, with the goal of minimizing its impact at the very top of the fraud funnel.
Since implementing our safeguards, we have seen fraud incidents mid-fieldwork drop from 19% to 2.7%, with a huge uptick in participants caught proactively higher up in the funnel. We encourage the wider research community to evaluate their best practices of mitigating qualitative fraud and continue this conversation, in the spirit of creating an industry that is firmly grounded in qualitative truth.
References
McKinney and Rosenzweig. 2022. “The state of data quality in research: a podcast interview with Cheskie Rosenzweig.”