By the Numbers: How researchers’ mental shortcuts open the door to online survey fraud

Editor's note: Sebastian Berger is head of science at ReDem GmbH. He can be reached at sebastian@redem.io.

Alongside the buzz surrounding artificial intelligence, the issue of data quality in online surveys has attracted more attention in the market research industry. The rise in both the scale and sophistication of fraudulent activities has contributed significantly to this concern. For instance, Kantar reports that researchers are discarding up to 38% of collected data due to fears over panel fraud and data quality.

Fortunately, increasing awareness of this problem has led to the emergence of numerous solutions. Efforts such as the Global Data Quality initiative have taken a broad approach to tackling the problem, while a growing body of research offers insights into potential remedies. At the same time, technological tools aimed at protecting data integrity are becoming more widely available. Given these developments, one might assume that the industry is well on its way to solving the problem. However, the reality is quite different. In fact, our data-cleaning efforts last year revealed that instances of fraudulent data had actually doubled.

From my work with market research agencies, fieldwork providers and research buyers, I have noticed that cognitive biases – mental shortcuts we as humans take – can make us more vulnerable to online survey fraud. To address this situation, it is crucial to critically challenge five prevailing mind-sets that leave us exposed.

1. The head-in-the-sand phenomenon

Too many industry professionals tend to ignore or downplay the risks of survey fraud to avoid confronting uncomfortable truths about data quality. This denial is often rooted in cognitive dissonance, where acknowledging the possibility of extensive fraud conflicts with prior assumptions or expectations. One such common belief is that fraudulent responses will cancel out over a large enough sample size, yielding an overall accurate result. While this might hold true if fraud rates were extremely low – just a few percentage points – in reality, with fraud rates commonly starting at 10% and above, this assumption no longer holds.

Another example is the argument, “Survey fraud has always existed,” which is technically true. However, past fraud detection methods – such as monitoring response speed, straightlining or using geolocation tracking – have become increasingly ineffective as fraudsters employ more sophisticated techniques. For instance, anyone can now easily create a customized GPT to automatically generate realistic survey responses, including colloquial language, short answers and errors in spelling, capitalization and punctuation. When coupled with software that inputs these responses while simulating natural typing behavior, such fraud becomes nearly undetectable. This evolution demands more advanced and proactive approaches to fraud detection.

2. The price-competition dilemma 

In the past, the drive for convenience, speed and affordability in online surveys has overshadowed concerns about data quality – a phenomenon known as outcome bias. Buyers have prioritized low costs and quick results, often at the expense of data integrity. On the provider side, strict quality controls have been seen as an added expense, reducing competitiveness in a price-driven market with tight deadlines. The absence of strict quality controls has turned online surveys into a commodity, creating opportunities for survey fraud. 

Given that primary research is conducted to inform crucial decisions in complex environments, where much is at stake, it's essential to follow the golden rule of risk management: never risk a lot to save a little. Tia Maurer from P&G illustrated this principle during her presentation at the International Quality Day, an event we hosted to explore survey quality challenges from multiple perspectives. She shared how poor data led P&G to launch an unsuccessful product, underscoring the costly consequences of poorly informed decision-making.

3. The “If I pay for it, it must be good” bias

This price-quality heuristic represents the flip side of the price-competition dilemma, fostering a misunderstanding between buyers and providers. It reflects the belief that paying for something guarantees quality, a mind-set especially prevalent with premium brands, known as brand loyalty bias. When buyers rely solely on price and brand reputation to trust survey data, without demanding concrete quality checks, they risk exposing themselves to survey fraud. This can lead to a dangerous assumption: “I’m paying them, so it’s their responsibility.”

As the saying goes, trust is good, but control is better. Buyers should demand transparent evidence of quality control measures down to the level of individual interviews, understanding which interviews were flagged or removed and why. This level of scrutiny is essential to safeguard data integrity and prevent costly missteps.

4. The “If I like it, it’s good” mind-set 

This fallacy, driven by the affect heuristic, leads buyers to question survey data quality only when the results do not align with their preferences (e.g., in pre-tests of various ad designs). Similarly, confirmation bias causes doubts about data quality to arise only when results contradict expected outcomes. Both biases distort objective decision-making by letting personal preferences, expectations and emotions dictate whether data quality is to be questioned. However, data quality must always be critically monitored, regardless of whether the results align with the buyer's expectations or preferences.

5. The abstraction dilemma

Professional survey fraud is invisible, intangible, complex, constantly evolving, difficult to localize, automated, highly technological and hard to define or detect. These factors contribute to its perception as abstract, making it challenging for market researchers to fully grasp the scope of the problem. This complexity often triggers oversimplification bias, a mental shortcut that reduces intricate issues into overly simplistic terms and solutions in order to achieve cognitive ease – a mental state of comfort that arises when processing easily understandable information.

Terms like “survey bots” and “click farms” are frequently thrown around in discussions of survey fraud, often lacking a deeper comprehension of their true complexities. A commonly used image in presentations and articles about survey fraud – depicting rows of people seated in front of computer screens in a dimly lit room – has shaped the perception of click farms. However, this image, popularized by the U.S. television series “Silicon Valley,” does not reflect the reality of modern click farms.

Today, click farms are more accurately described as phone farms, where multiple smartphones are interconnected and highly automated to carry out various online fraud activities, including surveys. These operations are typically run by an individual from home or a small group in a shared apartment, rather than in large, centralized call-center-like settings as the popular image suggests.

The oversimplification bias manifests itself also in the quality control measures used to combat fraud. Because survey fraud is so abstract and complex, many resort to simple solutions that provide a false sense of security. Reliance on tools like CAPTCHAs or trap questions is a prime example. These methods are easy to understand and implement but they fail to address the more sophisticated forms of fraud that occur today. This reliance on familiar, easy solutions is akin to rearranging deck chairs on the Titanic – a superficial fix that doesn't address the root of the problem. Tackling advanced technological survey fraud requires a comprehensive, multilayered approach that provides a 360-degree view of behavioral patterns and input data. Only by employing a wide range of quality checks and leveraging modern AI technology can survey fraud be minimized to a level where it no longer impacts results.

Significant gap remains 

While online survey data quality has rightly become a central concern in the market research industry, a significant gap remains between acknowledging its importance and taking meaningful steps to safeguard it. Despite growing awareness and the availability of advanced tools to combat survey fraud, the issue persists, showing that awareness alone is insufficient. To implement concrete, proactive measures, the five mind-sets outlined in this article must be actively challenged, as there is no viable alternative.

For instance, relying solely on synthetic data is not a solution, as these models are only as accurate as the data they are trained on. If fraud compromises that data, the results will be flawed – garbage in, garbage out. Likewise, blaming poor data solely on survey incentivization is misguided. While removing incentives may reduce fraud, it risks skewing the sample toward those with extreme views, undermining research integrity.

Additionally, attention must also be paid to data quality issues caused by inattentive or frustrated participants. Factors such as questionnaire length, fair compensation, screen-outs and thoughtful survey design are crucial in maintaining participant engagement and ensuring data accuracy. 

To effectively address these challenges, research buyers need to accept that their questionnaires may require changes, demand detailed insights into the inclusion or exclusion of interviews and be willing to invest in higher-quality results. Quality must take precedence over cost and speed! Market research agencies and fieldwork providers, in turn, should implement rigorous quality control measures and offer complete transparency in these processes. The era of black-box quality control is over – only those who offer full transparency and evidence of quality will succeed in the future.