Respondents lie and good ideas die
Editor’s note: Steve Richardson is director of communications for the Qualitative Research Consultants Association, St. Paul.
More than 500,000 focus groups were conducted worldwide in 2005, according to FocusVision. That’s a 2.2 percent increase over the previous year. Yet we read that focus groups are in decline, are not scientific and have no ROI. The reality is that the criticisms of focus groups have created some myths that have been perpetuated over the years - myths that hurt the qualitative research industry.
At the same time, the industry stands to learn lessons and can continue to improve itself by exploring key issues that relate to some customers’ and consumers’ perceptions of the discipline. This article will examine 10 often-heard myths relating to focus groups.
Let’s start with a story from the IT world. In early 2007, IBM rolled out a major upgrade to its Webconferencing and instant-messaging software called Lotus Sametime Version 7.5. The software sports enhanced features that make it easier for users to create and manage Webconferences, and to connect with other instant-messaging platforms in business settings, thereby relieving an administrative burden for IT managers.
This product rollout was a bit different for IBM. The company was used to talking primarily to IT and technical staff, but Lotus Sametime Version 7.5 is also a desktop function for non-technical businesspeople. Therefore, IBM needed insights from both groups that would allow it to maximize end users’ product understanding and IBM’s brand relationship with business consumers.
A combination of focus groups and in-depth interviews, conducted by Qualitative Research Consultants Association (QRCA) member Bernadette DeLamar of Business Strategy Research, elicited some crucial insights for IBM. For example, instant-messaging and Webconferencing technology have delivered significant benefits for customers and have resulted in a real acceleration of business activities. Many of the new capabilities tested, however, were difficult for ordinary business users to understand because of the IT jargon used to explain them. A user-friendly vocabulary that would be effective in advertising to end users was a major outcome of the focus groups. In addition, IT decision makers saw themselves as contributing real business value because they were introducing technology that would enable management to easily communicate and expedite decision-making. This insight became the basis of content for marketing messages that echoed the words of IT respondents who said that they were enabling their firms to "conduct business in real time."
"These critical insights helped us shape the product launch strategy and execution," says Carol Galvin, IBM’s senior segment analyst, who sanctioned the qualitative research effort. "There are moments in every good focus group that give us ‘aha’ insights that are just critical to communicating properly to our various customer segments."
One might assume that a company like IBM would care only about "measurable" metrics. But the company used qualitative insights (in addition to quantitative data) to develop and adjust a major product rollout - which leads us into our first issue.
Myth 1: Quantitative data is what really speaks to client decision makers due to measurable ROI.
In this age of measurement metrics, qualitative researchers often find themselves defending against the perceived lack of statistical significance of focus groups - and even apologizing for it. In their paper "Quit Apologizing: Rethinking the Limitations of Limitations," QRCA members George Balch of Balch and Associates and Christopher Herbert of The Insight Group/Focused Thinking explain, "In our work to dispel some of the myths which led to the perceived superiority of quantitative research over qualitative, we’ve collected many disclaimer statements in qualitative research reports from colleagues in the U.S. Many show a distinctively negative tone - ‘a note of caution,’ ‘not projectable,’ ‘not scientific.’ These statements warn readers and even apologize that the research is not, well, quantitative."
Here’s a case in point. A 2006 study conducted by Alan Janssen, health communications specialist at the Centers for Disease Control and Prevention (CDC), tested the response of pediatricians and infant caretakers to a new vaccine to prevent rotavirus - one of the most common illnesses in young children. Through various focus groups, researchers Dick Tardif of the Oak Ridge Institute for Science and Education and QRCA member Mark Herring of Market Directions Inc. learned something very important. While pediatricians embraced the idea of a new vaccine, the overall response to recommending the vaccine was tepid due to the history with a previous vaccine. The response from infant caretakers was also muted due to a general lack of awareness of the need to vaccinate, coupled with a somewhat blasé attitude toward rotavirus. Based on these findings, the CDC altered its communications approach to physicians. While not measured in quantitative terms, this research effort delivered significant ROI for the CDC.
The qualitative research community owes it to clients to help them understand (and sell up) the fact that qualitative research provides emotional reactions and deep-seated customer beliefs that cannot be extracted using quantitative methods. Clients don’t have to choose one over the other - in fact often both methods are used. A great example of ROI is conducting a qualitative study that significantly changes a project before a quantitative test, thereby preventing an expensive mistake. In today’s environment of "measure, measure, measure," we can’t lose sight of "understand" - and qualitative research is the discipline that helps us do that. The numbers rarely speak for themselves.
Myth 2: Focus groups are in decline.
The qualitative research industry has battled some high-profile criticism of focus groups, including one well-known chief marketing officer who declared "I’m killing all our focus groups" - and got a lot press for it. In fact for a time, criticizing focus groups became fashionable in some marketing circles. But guess what? Focus groups remain enormously popular - and for good reason. Focus groups are growing because the majority are conducted with a well-defined purpose, and they elicit truthful, deep-seated emotions from participants. In short, they are useful tools that perform an important marketing function that is hard to replicate.
Myth 3: Focus group participants are often people with too much free time on their hands.
Any step in the market research process can be done well or be done poorly - and the selection of focus group participants is no different. While underqualified participants may be motivated by the wrong reasons, properly selected subjects can offer amazing insights. So what does this mean for the qualitative research industry? First, don’t rush the process. Some clients hurry their way through research, including recruiting focus group participants, and we must insist on taking the proper amount of time to find the right people. In addition, we should do the best possible job of screening the suppliers who help us find participants. We must help guide them into using the proper screening techniques and qualifying questions and simply not allow the selection methodology to be incomplete, rushed or overly simplistic.
In his paper "What’s Wrong with Focus Groups and How To Do Them Right," QRCA member George Silverman of Market Navigation Inc. says, "Focus groups are not a process of asking small bunches of people for their up or down votes on products and their reasons to justify their opinions. They are a rich laboratory in which we get people to tell stories, react to each others’ stories, share feelings and express their values, passions, highest hopes and deepest fears in a variety of ways, often non-verbally or by other means that try to bypass their defenses." In other words, overly simplistic focus groups conducted with improperly selected participants is the real problem - not the actual discipline of focus groups.
Myth 4: Focus groups can be used to get quantitative data like that found in surveys and rankings.
Calling surveys and ranking discussions a "focus group" is a common external misconception thanks to many TV shows and news reports that talk about "focus group testing." Some clients that do little qualitative research may bring this misconception to a project. It is important for researchers to clarify that quantifying response to a marketing message, creative execution or advertising campaign is a useful tool for marketers but it is not qualitative research - it is not a focus group. Qualitative research gets at feelings, beliefs and emotional responses on a particular topic, not a quantitative rating. A researcher may ask participants to rank an idea or product on a scale, but that is usually designed as a first step to get the respondent to dig into an idea and elicit qualitative insights.
Myth 5: Consumers don’t (or can’t) express their true feelings in a focus group.
Focus group critics say that participants often give what they believe is the most rational response, or say what is socially acceptable in a peer setting, or what they think will make them look good or look smart. A participant’s responses may not always reflect his or her true feelings or attitudes. To access these below-the-surface feelings, experienced researchers use projective techniques to bypass respondents’ rational controls and let them express feelings they have trouble voicing in a standard question-and-answer format.
Myth 6: Focus groups alone are enough to unearth people’s opinions, motivations and feelings.
When well-conceived and -executed, focus groups in and of themselves can be a sufficient tool to satisfy qualitative research needs. But more and more often, top brands are using focus groups in conjunction with other qualitative (and quantitative) methods like in-depth interviews, online research, ethnography, dyads, mini-groups and telephone focus groups. Research buyers are recognizing the unique strengths that different qualitative tools offer, and are working more frequently with their research partners to incorporate multiple research tools to maximize customer insights.
Myth 7: Good ideas die in focus groups because consumers are not ready to accept new concepts.
QRCA member Judith Langer of Langer Qualitative addresses this myth in her book The Mirrored Window. She says: "Qualitative research offers researchers the opportunity to probe what underlies consumers’ reactions. If an idea meets immediate respondent rejection, a good researcher explores the reasons why, whether the product might fit a real need and what, if anything, could change respondents’ minds. Do the respondents lack knowledge? Are they anxious about newness or are they rejecting specific features? Is there a deeper resistance? The moderator can explore respondents’ reactions when given additional information or after learning of advantages they haven’t yet seen. The moderator can also explore potential changes in the product or service or in the communication."
Myth 8: The researcher doesn’t need extensive background on the research project or the unstated research objectives.
Some buyers hire a researcher as a "vendor" who can simply run a focus group by following discussion guides and writing a good summary report. That researcher may have a great reputation or be referred by a trusted source, so simply "doing their thing" fulfills the requirement for some buyers. However, clients who understand the real value of qualitative research know that treating the researcher as a partner - someone who understands the company, the scenario, the various influencers at play, and, yes, the internal politics - can deliver a far better qualitative research project. Clients may withhold information in the belief that too much background might sway the researcher or bog them down with information that doesn’t appear to impact the study. But just like any other trusted supplier, the qualitative researcher with a well-rounded understanding of the project will deliver far more value in the end.
Myth 9: Clients always know what they want out of research efforts and methodologies.
There are many savvy clients who understand the discipline, the methodologies and how to achieve desired results, but just as many are seeking consultation and expertise from their research provider in order to help them determine the best methodologies. And some clients may think they understand qualitative research, but they might appreciate additional insight on the proper methodologies. Being able to coach, guide and consult with clients is one of the most important pieces of expertise a researcher can offer his or her clients. A consultative partnership leads to better results, and moderators may need to educate their clients to think of them in that way.
Myth 10: It is important for the researcher to stick to the discussion guide because clients expect it.
In most cases, the discussion guide should be just that - a guide. Great focus groups use the guide to direct questioning and focus the session, but the real gems of insight are often gained by encouraging interaction, dialogue, digging deeper, understanding motivations, responding to what’s being said and improvising. Focus groups are one of the best research methods for this type of valuable interaction, and it is their flexibility that makes them so.