Real B2B research data goes a long way
Editor’s note: Nik Werk is the founder and managing director at Werk Insight
New research from the Insights Association shows that the average removal rate of online sample responses among research agencies is over 43%. For B2B sample, it is even higher. This alarming number sheds light on a pervasive problem with B2B market research. One that research agencies have been slow to address.
In the last year, I’ve spoken to many peers who have left sample providers and agencies. I’ve usually asked them a variation of this question: “If a client bought 100 online survey completes with B2B decision makers from a panel, how many of those are genuine?” Most said 10% or 20%.
B2B decision makers, like financial advisors or architects, are well-paid and busy. They are hard to reach and incentivize. B2B companies can struggle to get a 2% response rate to surveys sent to their CRM lists of existing customers. Yet research agencies claim to get hundreds of responses in a week from these experts for minimal incentives.
The truth is that most market research agencies have no idea who is taking their online surveys. The industry claims that data quality issues can be combatted with screening and validation techniques but there are fundamental flaws to those methods.
Until these issues are resolved, insights drawn from online surveys are questionable at best. Instead, we must turn to alternative methods of data collection where respondent identity is not doubted.
How online survey data is collected
Market research agencies usually outsource most, if not all, of their quantitative data collection to third-party sample providers. The respondent data is anonymous for compliance reasons, but that also masks the respondent’s identity from the agency. This makes it harder to detect if people who don’t have the necessary qualifications filled out the survey. It’s not always a clear-cut case of fraud – it might be a finance graduate who says they are a financial advisor.
The current industry approach is to tackle the issue of data quality with screening questions and data quality checks. This could include technical questions in a survey that only a “real” financial advisor could know or tracking the respondent’s behavior and geolocation to detect suspicious activity like responding to a survey multiple times from the same location. The most rigorous quality checks tend to happen after the data is collected. Any survey response that “looks off” is rejected by the agency and replaced by their sample vendor.
This is like sticking a band-aid on a gaping wound. Research agencies hope that it will quell clients’ concerns and stop them from asking questions about the quality of the data.
Weak data verification: Screening methods are not always secure
The high number of responses being rejected from online surveys is a red flag. Agencies still don’t know if the remaining data that pass these checks is real. They are not asking respondents to verify their identity face-to-face with ID documents or business e-mail addresses. In this way, respondent anonymity serves as plausible deniability for agencies in front of their clients – even while they reject large volumes of survey responses from their studies.
Screening questions and data validation methods have clear loopholes. ChatGPT can provide convincing responses for any technical industry questions used for screening, geolocation can be masked with VPNs and digital fingerprints can be washed without requiring much technical expertise.
With AI, agents and bots can now replicate human behavior and expertise at scale. This will make it much harder to detect fraud going forward.
Lack of accountability
The rise of sample intermediaries in recent years is compounding the problem. Agencies hire a sample provider to field surveys, who then outsources the project to multiple third-party providers. Online sample marketplaces allow sample providers to easily share access to pools of respondents with each other. A survey might be fielded by a handful of sample providers with varying standards of data validation. This creates an even greater distance between research agencies and respondents.
Everyone in the chain is incentivized to meet the target number of completed survey responses. Many research agencies turn a blind eye because that is easier than reckoning with the consequences of the data not being real. That’s clearly the case even when large numbers of responses are being rejected. The issue with data quality is not limited to “a few bad apples” – it is systemic.
Regaining client trust in data quality
Until the industry commits to stronger action on reducing widespread fraud, online surveys will fail to provide meaningful B2B insights. In the meantime, the best way for agencies to gather genuine data is through computer assisted telephone interviewing or video interviews. This way, an expert’s identity can be verified face-to-face using ID or a business e-mail address to send the meeting invitation. This approach costs more and fieldwork takes longer. Agencies can offset some of these disadvantages with AI, which is already bringing efficiency to the research process in other ways.
The enduring value of insights from real data
It helps that clients are waking up to the data quality issue in online B2B surveys. Some will continue down the path of ever-faster, ever-cheaper data – and there will always be research agencies willing to hold their nose and take the money. However, those who recognize the value of real insights should demand higher-quality sources and stricter verification methods from their agencies and sample providers.
In the new era of AI, research agencies will only remain relevant if they justify the value of primary data. High-quality data from real respondents is the foundation for insights that have a real impact on critical business decisions. Brands that commit to that in their market research will have a competitive advantage like never before.