Editor’s note: Mark Goodin is president of Aaron-Abrams Field Support Services, a Las Vegas research company. This article appeared in the December 8, 2009, edition of Quirk's e-newsletter.
The time has come for the research industry to upgrade the quality of its product by cleaning out professional respondents. Professional respondents are not a one-off occurrence, and their presence threatens to contaminate market research and call its credibility into question. Failure to raise awareness about the extent of this problem is preventing it from being dealt with. So why is there no outcry?
The answer is twofold: a hesitancy to publicly admit weakness and, of course, cost. It has to get worse before it gets better, and many companies are reluctant to deal with the initial fallout - even when the end result benefits data quality and research’s image. Fortunately, the problem is not without a solution; there are several steps researchers can take to face the issue head-on.
Cast a bad light
Most researchers will candidly agree that talking about professional respondents could cast a bad light on qualitative research. That makes sense: If researchers speak out, then they are saying that there's a problem with the methodology - essentially shooting the industry in the foot. End-user clients might shift away from qualitative methodologies, which could potentially affect many careers.
Companies that recruit respondents for internal or external use also refuse to talk publicly about this problem. Admitting that dishonest respondents inhabit their databases or pass through their doors might result in clients shunning them as well. That also makes sense: Why would researchers want to hire any firm that openly admits to having problems with respondents?
Respondent recruiting is big business. Keeping a steady supply of respondents flowing in and out is among the top priorities for recruiters. And for researchers, one of the top priorities is getting bodies at the right price. The problem with eliminating professional respondents - and the ease with which they can be recruited - is that it would add delays and increase costs.
However, ignoring professional respondents or choosing to work around them has caused this problem to escalate. And rewarding these individuals with payment has drawn cheaters, repeaters and other problem respondents to qualitative research.
Control this epidemic
Things weren't like this prior to the advent of the Internet, which has made it increasingly difficult to control this epidemic. The Internet has given respondents easy access to sites like Craigslist and mass-registration pages on suppliers' Web sites and the ability to easily seek and sign up for market research studies. What's more, a plethora of information is now available online that explicitly coaches respondents on how to cheat and lie their way into research.
While many research organizations have successfully implemented security procedures that have reduced the magnitude of the professional respondent problem, others have not. The good news is this: There are ways to improve the situation. Following are some proven tactics for clearing professional respondents and their networks out of qualitative research.
Shut down entry portals. Researchers and recruiters must remove mass-registration pages from their Web sites. Although these pages are an easy and inexpensive way for suppliers to add respondents to their databases, they also make it easy for cheaters to find researchers and gain access to databases without being challenged. Respondents must be spoken to individually before being added to a database!Much can be learned about potential respondents and many problems can be avoided via this simple, old-school step. Allowing respondents to complete registration questionnaires without being spoken to opens the door to problem respondents.
Stop relying on Craigslist and similar sites. For the time being, anyway, these places are populated with problematic respondents. When ads are placed on Craigslist, responses often return within seconds, which indicates that respondents are not innocently stumbling upon ads for research. Instead, it’s likely they're being automatically alerted to them. These types of respondents want to participate in as many projects as possible, regardless of qualifications or restrictions. What's more, these respondent sources make it nearly impossible to track respondents who know each other. Knowing the who-knows-who aids in identifying problem respondent networks.
Discard existing databases. Respondent databases with a history of cheaters must be scrapped. Yes, some honest respondents will be lost, but that's the price database owners ultimately face for allowing undesirable respondents to infiltrate their records. Cleaning up the professional respondent problem requires recruiters to return to seeking respondents. Unsolicited call-ins are the hallmark of professional respondents and do not belong in the active-respondent file.
Return to cleaner respondent sources. Telephone books and directories, client and third-party lists and company Web sites are a few examples. Critics argue that using these old-school sources is cost-prohibitive, and that they fail to reach hip, fresh respondents. True, these recruiting methods can be more costly than recruiting respondents who seek research opportunities, but there is a quality versus cost issue for researchers to consider. And as far as reaching fresh respondents is concerned, social networking sites are filling with the same problem respondents who crowd sites like Craigslist. Broadcasting research opportunities to an unfiltered population is bad practice. The new priority for today's recruiters and research buyers should be to avoid coached and deceptive respondents at any cost.
Verify identities with background checks. The most powerful "burglary" tool that professional respondents currently possess is the Internet. It helps them gain entry into recruiter databases and market research studies and teaches them how to game recruiters.
However, there are ways to turn the power of the Internet against professional respondents: by revealing information about respondents that they didn't intend for recruiters to know. For example, simple searches of respondents' names, company names, phone numbers and e-mail addresses can corroborate the personal information they have provided; zero hits on a search should be considered a red flag.
Another solution that, if implemented industry-wide, would go a long way toward eliminating professional respondents is a standardized permission waiver allowing recruiters to perform background checks on prospective respondents. Applying for a job, a mortgage, a car loan or a leadership position in a volunteer organization requires this sort of waiver, and people sign them readily every day (provided they have nothing to hide). Recruiters need to entitle themselves to the same level of confidence when selecting respondents by verifying identities and, when possible, product or service usage or ownership.
Inevitabily, background checks will reduce respondent participation. Some legitimate respondents will be unable or unwilling to provide required proof. Others will be impostors who don't want to reveal their game. Either way, in return for cleaner respondents, researchers must understand that recruiting may require additional time and money.
Go high-tech with respondent verification. Similar to systems used by health clubs and membership warehouses, focus facilities can issue picture IDs to respondents. When respondents arrive, their ID cards can be scanned and their identities verified. Networking individual databases into a centralized catalog in each market would be a powerful deterrent against repeaters and respondents who use multiple identities.
Require proof of use or ownership. Respondents should be required to provide proof of use or ownership of the product or service being researched - in the form of documentation or the actual item itself. When practical, this should be a precursor to participation.
Stop paying hush money. Do not reward respondents whose deception has been discovered. Stop paying these people! Remove them from your research immediately. Tell them that they are not being paid and let them know why. If a respondent causes a commotion in your offices or a focus group lobby, for example, call law enforcement immediately. Word will quickly spread among professional respondents that researchers no longer tolerate or reward cheaters. Extreme measures like these are needed to put an end to cheaters and preserve the livelihood and credibility of researchers.
Stop instant payments. Discontinue the practice of paying respondents on the spot. An alternative approach is to require mailing addresses for all respondents with the understanding that incentives checks will be mailed. This offers several advantages.
- Research venues are safer, since no cash is kept on hand.
- Cheaters can no longer count on receiving immediate cash rewards.
- It deters respondents from causing a scene or using intimidation to take the money and run.
- It increases chances of obtaining valid mailing addresses for serving subpoenas, if need be.
- It eliminates having to confront problem respondents on-site. If a respondent is suspected of deception, further investigation can be conducted to determine whether that person should be paid.
- Researchers aren't charged incentives for respondents who don't qualify.
Some believe that genuine respondents will not be interested in participating if they are not paid on the spot, but the fact is many people are accustomed to getting paid after they perform services. Several current methodologies compensate respondents at a later date. Respondents who refuse to accept the new payment terms may be doing the industry a favor by outing themselves as cheaters.
Prosecute cheaters. Take legal action against respondents who use deception to receive compensation. These respondents are stealing from you! Saying that it's not worth it for a double-digit incentive is the kind of thinking that allows professional respondents to thrive. Legal proceedings will send a message to cheaters that deception is no longer tolerated.
Raise recruiting incidence. Researchers must simplify the recruiting process and everything that goes along with it. Increasing recruiting incidence makes it possible for recruiters to find qualified respondents without having to scrape the bottom of the respondent barrel. Additionally, raising recruiting incidence often reduces incentive amounts, which helps to remove the lure that attracts professional respondents.
Create an industry advocacy. Establish a member-supported professional organization to fight professional respondents on a global scope. Its duties would include:
- prosecuting individuals and companies that use deceptive practices;
- removing electronic and print material that instructs people how to use deception to get into market research studies;
- stopping improper advertising;
- establishing and enforcing industry-wide standards for training and licensing of focus facilities and respondent recruiters;
- providing continuing education components and mentorship; and
- providing mediation for recruiter and client complaints and grievances.
Sorely needed
Yes, implementing these measures will add extra costs, but a transformation in the way respondents are recruited and managed is sorely needed to secure research’s credibility and accuracy. Arguing that these changes are unreasonable and impractical is tantamount to saying that qualitative research is the big joke that professional respondents take it for. A great first step is for everyone tasked with recruiting respondents to receive the right tools and education to become more sophisticated than tomorrow's professional respondents.