Editor's note: This article appeared in the January 25, 2010, edition of Quirk's e-newsletter. The following article is a response from Judy Langer, president of New York-based Langer Qualitative, on behalf of the Qualitative Research Consultants Association (QRCA) Field Committee, to Marc Goodin's article "No more Mr. Nice Guy: Professional respondents in qualitative research" in Quirk's December 8, 2009, e-newsletter, in which Goodin shed light on the problem of professional respondents in qualitative research and offered several steps researchers and recruiters can take to ferret out the cheaters. Langer's response is as follows:

The quality of qualitative has always boiled down to two key components: the researcher's work and the fieldwork. A moderator may do outstanding work on a study design, interviewing and analysis; however, if some participants have purposely misrepresented themselves, the value of the research is thrown into question. People who lie about their characteristics, behaviors and/or involvement in research studies have been an industry concern almost as long as qualitative methods have been used in market research. These people, often labeled "professional respondents," attempt to wrangle their way into studies by deceptive practices. They are dishonest about how frequently they have participated in qualitative studies or, worse, pretend to be something/someone they aren't.
 
Goodin's article states that the problem of cheaters has become an "epidemic" that "threatens to contaminate market research and call its credibility into question." He asks, "Why is there no outcry?" His answer is that, "If researchers speak out, then they are saying that there's a problem with the methodology - essentially shooting the industry in the foot." One of his suggestions is to "create an industry advocacy."
 
In fact, the industry advocacy organization already exists. With nearly 1,000 members in 30+ countries, the Qualitative Research Consultants Association (QRCA) has been at the forefront of efforts to raise quality standards for almost 30 years. QRCA promotes education of qualitative research consultants and clients, and collaborates with other industry organizations.
 
From its beginning, QRCA has worked on the problem we call "cheaters and repeaters." It is probably impossible to quantify the size of the problem or to know if it has grown worse with the advent of the Internet. Anecdotal evidence and some qualitative research consultants' own experiences do show that the problem exists to some extent. More importantly, the QRCA believes that whatever the size of the problem, it is too big. (Note: Cheaters and repeaters are not unique to qualitative, the issue arises for quantitative research as well.)
 
QRCA, often in conjunction with the Market Research Association (MRA), have published several best-practices and guidelines documents, available on our organizations' respective Web sites (www.qrca.org and www.mra-net.org).These documents can be used as resources regarding managing the challenge of cheaters. For example, the respondent identification best practice is a simple yet effective policy of facilities requiring that respondents in non-list consumer studies show official IDs (i.e., government-issued ones like driver's license and passport or institution-based ones like student IDs). Just informing potential respondents of the policy undoubtedly scares some fakers away. Yes, hardcore cheaters can go to the trouble of creating fake IDs, but many facilities use their databases to track and cross-reference names, demographics and contact information in order to catch these people.
 
The Qualitative Handbook, available on the MRA Web site, offers guidelines and recommendations on quality fieldwork. Documents include topics such as screeners/rescreeners, database management, e-mail recruiting guidelines and the ID program. QRCA soon will be releasing a Field Committee report on its study among facility recruiters concerning respondent cooperation and the cheater problem. Our organizations have recently reinstituted a joint committee to find new solutions to fieldwork issues. We are also working with the Marketing Research and Intelligence Association (MRIA) in Canada to learn more about methods they are employing for reducing cheater/repeater incidence.
 
Most qualitative fieldwork facilities, we believe, maintain high standards for recruiting. Qualitative consultants and clients may not be aware of some behind-the-scenes safeguards they use to ensure respondent quality. Facility measures include:

  • Flagging cheaters/repeaters in facility databases. Databases not only help facilities to recruit but also to monitor frequency of participation and avoid dishonest respondents. The fact that people are in a database does not mean they will be recruited for a study. Cold-calling samples are very impractical in today's world since many people contacted are suspicious and unlikely to participate - some may even be the professional respondents we are trying to avoid.
  • Carefully screening and recording call-ins from people who volunteer to participate in a study. This separates legitimately-interested consumers from cheaters.
  • Using multiple rescreening steps in confirmation calls and at the facility to check the consistency of respondents' answers. While some inconsistencies can stem from memory problems or unclear screeners, wild divergences, especially on basic questions like demographics, are suspect.
  • Requiring respondents to present government-issued identification for non-client list consumer studies, as mentioned above, and business cards for business-to-business studies. Current employment status and other characteristics can sometimes be checked on social networking sites and through Internet searches.
  • Asking respondents to bring in proof of product ownership and use, where appropriate. For example, in pharmaceutical studies, respondents can bring in a bottle of a prescription medication.
  • Limiting the nature and extent of e-mail blasts, the use of social networking sites and Web sites like Craigslist to find respondents. These methods should only be used on particularly difficult recruits and with client approval. In these instances, questions should be worded to disguise the exact specifications for respondents. Respondents who make it through this initial step should be screened by a live recruiter on the phone and through the other methods outlined here.
  • Closely supervising and monitoring recruiters to make sure they follow protocols; scheduling regular refresher courses.
  • Rescreeners can include a statement saying that the respondent has been and will be honest in the research. Signing the statement does not have legal standing if the respondent does lie, but it does serve to emphasize that fieldwork services are serious about wanting truthful respondents.
  • Internally posting known cheaters' names and photographs to help facility staff members identify and exclude them if they show up for an interview.
  • Sharing information, within the constraints of privacy laws, about inappropriate or unqualified respondents with others in their market. Several facilities and facility associations already do this.

Importantly, qualitative research consultants and end clients share responsibility for reducing the number of professional respondents. Here are a few steps they can take:

  • Writing screeners so professional respondents cannot easily figure out how to game the system. Recruiters participating in the QRCA Field Committee study mentioned above recommend that key questions be asked in an open-ended, unaided way. Having interviewers read a list of answers can inadvertently help respondents guess how to qualify (e.g., choosing a higher frequency of product usage). Such questions can be prefaced with a statement like, "We're looking for a mix of people who have and have not used a product" - even when only one type qualifies for the study.
  • Writing effective self-administered rescreeners to weed out respondents who don't qualify or who misrepresented themselves earlier. These questionnaires should ask just a few questions, using the same wording as the original screeners. Having respondents fill out the forms themselves prevents coaching by facility staff (whether inadvertent or not).
  • Selecting fieldwork services the researchers/clients have had good experiences with and/or have solid professional reputations. (Choosing a facility on price alone is never a good strategy.) When qualitative researchers need a different or new facility (e.g., when they go to a market they haven't used before), colleagues' recommendations can, of course, be a useful guide.
  • Spending time asking facilities, especially ones not previously used, about their practices (e.g., do they have in-house recruiters or not; what steps do they take to validate respondents' identity and behavior; do they use respondent IDs on consumer studies; etc.).
  • After the interviews, telling facility managers about any respondents who raised concern so they can be flagged in the database for closer watch in the future.

The QRCA continues to look into better ways of reducing the number of professional respondents who participate in research studies. We encourage QRCs and facilities to follow best practices and join us in raising the caliber of qualitative fieldwork. We welcome your suggestions at fieldcom@qrca.org.