Keep the quality in qualitative research
Editor’s note: Judith Langer is senior vice president, qualitative practice, at GfK NOP, a New York research firm. Carol Stuckhardt is director of custom research, Hearst Magazines, New York.
Here’s a statement that virtually all (maybe all) qualitative research practitioners and clients would easily agree with: the qualitative research consultant (QRC) is a critical element in the success of a study. If this seems obvious, it is surprising that QRCs are often thought of and treated as a commodity, and that substandard work is often tolerated by clients. Over the years, we have seen or heard of many examples of work that ranges from the merely mediocre to the truly bad. (And, happily, many other examples of excellent work.)
What accounts for clients accepting substandard work? Some clients, we believe, simply do not have enough research expertise overall or enough familiarity with qualitative research specifically to effectively evaluate QRCs. As a result, they:
- do not realize that they should demand a higher level of work;
- have unreasonable expectations about QRCs, confusing skill with showmanship;
- buy (almost totally) on the basis of price, thinking QRCs are pretty much the same;
- focus on qualitative “techniques” and forget about the critical role that the QRC plays, regardless of method.
How much does this matter? For argument’s sake, if clients accept or at least tolerate so-so quality, maybe it’s okay - especially if they get their work at bargain prices.
The problem is that it does matter. When the research is done poorly, it may not yield insights that help the client and a good deal of money goes to waste. From a broader research industry perspective, the focus group - and by extension all qualitative research - gets a bad name as shallow or false.
What clients should expect of a QRC starts with their view of the researcher’s role. “Moderator” is a common term, one we often use ourselves, but it refers just to the interviewing portion of the job. In the case of focus groups, which U.S. clients typically observe from behind a one-way mirror, the interviewing is the most visible part of the job, of course. For the qualitative research process to be successful, however, a QRC needs to be a true researcher involved in all three stages of a project:
Phase 1: Pre-interview - designing the research sample and method, preparing the materials for recruiting (screener) and interviewing/observing (topic guide), supervising the fieldwork.
Phase 2: Interview/observation - engaging participants/respondents through questioning and observation.
Phase 3: Post-interview - figuring out the meaning and implications of what the researcher saw/heard, reporting/presenting it to the client.
For ease of reading, “focus groups” in this article refers to in-person group discussions. However, we believe that our recommendations are equally true for other qualitative research methodologies, including in-depth interviews, observational interviews and online bulletin boards.
To learn what not to accept, we gathered some real-life examples of low-quality or problematic qualitative research practices. All the instances cited come from other clients and QRCs we have talked with, as well as our own observations. While some of the instances cited may seem extreme, we promise that every one is true!
At each point in the qualitative research process, you know there are problems if a QRC does any of the following:
Phase 1: Pre-interview problems
Beware the QRC who:
1. Announces he/she “only moderates,” and does not manage the project process or write reports.
2. Proposes novel methods that seem strange, cringe-inducing or superficial - defeating the purpose of qualitative research. (Does a hall of 100 respondents speaking out talk show-style sound in-depth to you?)
3. Has a solution that is “always” (or even almost always) used rather than a customized approach to the client’s problem.
4. Takes your RFP or topic guide draft without asking detailed questions about the study’s purpose, goals and background.
5. Suggests (or agrees to) an overly ambitious agenda with too much material, too many issues/questions/exercises/visuals. With 30 concept statements, for example, respondents can only do a quick thumbs-up/thumbs-down, not discuss their feelings about the ideas.
6. Ignores fieldwork, assuming it will “take care of itself” without close supervision.
7. Does not inform the client about recruiting problems until the last minute, if at all.
8) Makes changes in fieldwork without obtaining client approval or even informing the client (e.g., respondent specifications, raising incentives, telling respondents who the study sponsor is).
Phase 2: Interview/observation problems
During the introduction/warm-up, beware the QRC who:
1. Is long-winded, giving an overly detailed explanation of the process and ground rules; wastes precious time; is boring; is off-putting. (Do respondents really need to know how many observers are in the backroom and what they are eating? Do they really need to be told that if they need to go to the restroom, they should do this one at a time?)
2. Offers a “too much information” introduction of him/herself (marital status, number of kids, upbringing, hobbies, etc.). The best moderator is usually a neutral party. A good moderator establishes rapport without divulging his/her biography.
3. Asks respondents their names but does not give his/hers. While personal details should be avoided, some information helps establish a relationship - and introducing oneself is simply good manners.
4. Uses deceptive warm-up tactics/tricks (e.g., pretends a respondent did not show up and asks the group to describe him, then announces this person never did exist). This makes respondents feel fooled/tricked, and undermines trust.
During the course of a focus group, beware the QRC who:
1. Does serial depth interviews in a focus group, circling around the table repeatedly, calling on respondents one by one. This prevents or squashes interaction, a key reason to conduct focus groups in the first place.
2. Dwells on “interesting” subjects irrelevant to the study purpose.
3. Ignores timing - starts the interview late without informing respondents; keeps respondents beyond the promised end time without asking their permission. This shows a lack of respect for participants, breaks the moderator’s implicit contract with them, and risks harming rapport and good participation.
4. Does not invite quiet respondents to enter the conversation; or puts them on the spot. (“George, you haven’t said anything yet” is not a good way to draw out a quiet person.)
5. Repeatedly calls on the same respondents, ignoring the others. (We have seen male moderators who only ask/allow the men to speak.)
6. Deceives respondents by doing things like planting a ringer to stimulate discussion or, worse, to sell the client’s product. Such practices are just plain unethical.
7. Acts like the star of the show, talking more than the respondents, playing stand-up comedian by telling canned jokes. The respondents - not the moderator - should be the center of attention.
8. Talks to respondents in a demeaning or sarcastic way, shuts them off harshly. Aside from being rude, from a research perspective, this demonstrates a lack of respect for the respondent and damages group rapport.
9. Embarrasses respondents through personal or inappropriate questions/exercises. (If asking respondents to act like their dogs at mealtime makes respondents squirm, it probably isn’t a good idea.
10. Lacks energy; speaks rarely or in a soft monotone. This, in turn, lowers the energy level of the group.
11. Is a passive non-leader - lets the group get out of hand by allowing respondents to talk endlessly, simultaneously or on tangents, with no attempt to bring them back on track.
12. Does not deal with difficult respondents who talk over others or are argumentative/hypercritical/hostile, etc.
13. Marches through the topic guide - reading it verbatim to respondents (staring at the paper, rather than making eye contact), moving on to the next question rather than following up on respondent comments.
14. Does little probing.
- Does not follow up on key issues (e.g., asking why a respondent would “never” use the client’s brand). One moderator told his client that he asks the questions on the guide and thinks about what the answers mean afterwards.
- Uses only basic, boring probes (“What do you mean?” “Anything else?”) rather than delving deeper.
- Asks the same question the same way over and over until respondents protest or lose interest.
15. Uses a focus group as a data-gathering venue (e.g., announcing in the session that “Seventy-five percent of the room thinks x,” which implies that the group is a mini-survey).
16. Dresses inappropriately for the particular respondents in level of formality, hipness/sophistication, sexiness. Looking strange or superior to respondents can prevent rapport-building.
17. Shortens the length of depth interviews over the course of the day, thereby shortchanging the client.
18. Acts in a weird/distracting way (e.g., using the moderator chair like a scooter to roll around the room).
Phase 3: Post-interview
During the post-interview phase, be wary of the QRC who:
1. Leaves out or fails to address the client’s objectives in the final report/presentation of findings, and just writes about whatever interests him/her.
2. Submits a poorly written report that:
- is boring/dry, ungrammatical, unproofed, etc.;
- is semi-quantitative in style or shows tables/numbers (“three respondents,” “25 percent said”) - this is qualitative research, not numbers-counting;
- presents only what respondents said without interpretation and analysis (what does it really mean?);
- relies on verbatims to tell the story, is quote-heavy with little text to report/explain;
- lacks analysis of marketing implications;
- recommends naïve, impractical and/or extremely expensive ideas that conflict with the client’s business realities.
Talent and training
Based on the examples cited, we hope you will agree that high-quality qualitative research takes talent and training. Like many things in life, good qualitative research may (and perhaps should) look easy, but that doesn’t mean it is. And, no, not everyone who “enjoys talking with people” can do it; analytical abilities are needed throughout the process. The best qualitative research is truly exploratory, taking advantage of serendipity, and is not mechanical or rigid.
How should a client find and select the right QRC? The choice of the QRC is the critical first step in the process. For clients who do not already have a researcher they have worked with successfully in the past, here are some suggestions:
- Hire seasoned professionals you trust, are comfortable with and/or who are recommended by people whose judgment you trust.
- Choose QRCs who are skilled at qualitative research and understand marketing issues.
- Don’t overpay, but recognize qualitative research is not a commodity, that QRCs are not all equally good.
- Expect the QRC to be responsible for the entire project - working with you at all stages.
- Expect the QRC to be focused throughout on the true objectives of the research, tailoring all materials and the interviewing approach to those goals.
- Choose good thinking over novelty for its own sake - try new techniques only if they make sense for the objectives and you are willing to take the risk.