Try getting more involved in the process

Editor’s note: Stephen H. Turner is president of Chicago research firm fieldwork, Inc.

In the spring of 2005, Malcolm Gladwell delivered the keynote speech to American Association of Advertising Agencies, much of which was based on his research and thinking as found in several New Yorker articles and elegantly detailed in his book Blink. As was widely reported since, he criticized focus groups for delivering information so misleading as to be counterproductive.

In a Newsweek article (November 14, 2005), Cammie Dunway of Yahoo, Inc. said that she is abandoning traditional focus groups in favor of “immersion groups,” which she believes will be less likely to produce “false positive” responses.

In this context it is valuable to attend carefully to Gladwell’s closing remarks from his speech:

“Market research, when it is observational or when it is interpretative, is profoundly useful. But those are two critical things. They require the intervention of the person conducting the research. They require the findings that are gathered are considered, and thought about, and processed and interpreted. Back in the 1950s, most of the major advertising agencies on Madison Avenue employed Freudian psychoanalysts for this precise reason, and you don’t see that anymore. I think that’s a big mistake.”

As a focus group moderator with 40 some years of practice, I would be the last person to suggest that respondents in a focus group environment are forever disposed to expressing empirical or behavioral “truth.” I’ve done too many groups with TV addicts who, with straight faces, avow a deep distaste for mindless programs and the advertising they are sandwiched between only to have them follow their diatribes with comprehensive and entertaining descriptions of both. I have listened endlessly to folks railing about the nutritional depravities of fast food minutes after observing them depositing crumpled McDonald’s bags into the trash as they filed into the room. I have attended countless sessions where group participants expressed righteous affinity for products they neither needed nor wanted.

But I contend that my clients are rarely fooled and almost always enriched by these apparent anomalies. They, and I, have come to understand that such empirically false expressions have social meaning - they carry important messages regarding what people hope to project about themselves in social settings. They may be false in terms of their behavioral components but they can be extraordinarily valuable in terms of social import.

Not misleading

In Blink, Gladwell talks about two Japanese words that we don’t have (but may need) in English - one for something that people profess to be true even though they understand that it is, in actuality, false; the other for something that people assure each other is hoo-hah even though it’s “known” to be true.

My sense is that much of what is “misleading” in focus groups isn’t (or at least shouldn’t be) misleading at all. It’s part of the urban myth syndrome so aptly captured by those two Japanese words - people expressing things they want others to hear them expressing because it says something about them socially. The truth is, almost everyone in the group, and certainly a seasoned moderator, knows implicitly that the speaker is expressing an aspirational truth that happens to be empirically false (and may always be). We may not have codified the phenomenon with a word as the Japanese have but we understand and we don’t think of it as misleading - simply an expression of a different type of truth.

Indeed, I have found over the years that most people hold tight to at least two sets of truths about themselves and their lives and they are quite capable of espousing both in the same breath even if they appear to be logically inconsistent. One is based on a set of generalizations regarding one’s social persona (what kind of person I am). The other has to do with one’s literal behavior that at any point in time may or may not be in concert with one’s projected social self (what I do).

When people in focus groups describe themselves in terms of social truths, even though they may stray a good distance from empirical reality in their lives they don’t generally think of those wayward descriptions as being untrue. They aren’t, in their minds anyway, lying. They’re telling others in the group what sort of person they think of themselves as being and wish to be thought of as being (whether they are is another matter). They are projecting an inner image and public persona.

Attentive to discrepancies

In my years as a focus group moderator, I have patterned my interviewing and analytic style to emulate mentors who were sensitive and attentive to apparent discrepancies between social and empirical “truth.” Indeed, a good deal of the time I spend in groups involves winnowing out those discrepancies and calling respondents’ attention to them.

The results can be very revealing. When does behavioral reality follow and when does it fly in the face of social reality and why? What are the valences that people ascribe to different levels of “truth” - in bringing a product to market how much weight should be given social and empirical issues? And, what might be done to position the concept so that it doesn’t offend the social ideal?

Most people tend to veer toward social (projective) truth when they are talking about the indefinite past. Ask someone about his or her “usual” or “typical” behavior and you’re very likely to get a summary that has little to do with empirical truth. People simply aren’t very good about developing a precise measure of their own central tendencies so they fill in their ideal ones.

This is even more problematic when people speculate about their future behavior. Such projections are, by necessity, based on one’s sense of what’s typical or usual, so empirical precision is even more remote. Most people tend to tint their expectations with a brush full of idealistic color.

Accordingly, the primary value of focus groups should not rest on respondents’ ability to predict the success of a product as it is presented in the group. They are great for exploring the underpinnings of acceptance or rejection; for suggesting different ways to design, position or communicate about the product so that the specific elements that add to or detract from the concept are optimized to give it its best shot. They are excellent at finding a fatal flaw or a heretofore unanticipated goldmine in the basic idea or the way that idea is presented.

This having been said, a host of observations can be parsed from a focus group response that will get you very close to empirical, behavioral truth. The energy with which a concept is greeted is one example. The ease with which people lace the concept into the context of their actual behavior is another. While the evaluative component of group discussions can be quite misleading, the logic people use to support their evaluations can be most helpful. It is attentiveness to phenomena such as these that allows a truly proficient analyst to connect “social” and “behavioral” truths, thereby making qualitative research actionable.

The point is that focus groups don’t have to be misleading. In fact, though I may be ambushed by my colleagues for this blasphemy, I have been convinced over the years that if a quantitative study and a qualitative one fundamentally disagree over a given finding, it’s usually the quantitative work that’s at fault. This is largely because focus groups sacrifice reliability in favor of validity while quantitative work sacrifices validity for reliability. And, when you have a problem with quantitative findings it’s usually a validity issue.

I concur with Gladwell’s contention that marketing research becomes profoundly useful when it incorporates a strong interpretive component and can be counterproductive when that component is missing. Good qualitative research, in my mind, becomes useful only when it’s accompanied by careful interpretive thinking.

I fear that dismay at the results in the absence of interpretive thinking is the primary reason behind this latest cycle of focus group bashing. All too often researchers and clients alike are taking responses in focus groups literally without regard to what types of truths are being portrayed. I find this amazing. Are there any other situations in which intelligent people expect to be able to take the word of strangers at face value?

Lazy clients

And now I’m really going to rev up the lynch mob. I contend that the problem is as much or more the fault of lazy clients as it is of mediocre moderators. Years ago, when “Freudian psychoanalysts” were the qualitative practitioners of choice, clients were interested in tracking the logic by which qualitative “findings” were revealed. They demanded strong, narrative reports that built a case and they actually read them. They wanted interpretation, digested it, and were prepared to evaluate its worth.

Today that sort of client/analyst relationship is a rarity. Most focus group studies culminate in a topline set of observations summarized in a few PowerPoint bullets delivered a day (or less) after the last group. Clients want answers unfiltered by anyone - moderator, psychoanalyst or marketing consultant. They want respondents to do something that’s impossible: accurately report and predict their own empirical behavior and attitudes. And when that fails, they say that there’s something wrong with focus groups!

What better way to assure that erroneous observations make it to the boardroom and to be condemned later because they were patently wrong? If you’re in a hurry to summarize hours of loose conversation, if you have a client who doesn’t want to be bothered with wading through the logic, you’re going to regurgitate literal rather than interpreted observations - you’re going to report social truths as if they were empirical truths.

Pressed for answers rather than reasoning, most moderators simply cannot afford to take the time to build the logic necessary to support a reasonable prognosis. You’re simply going to “take their word for it.”

Of course, moderators aren’t free of culpability either.

It’s an easy business to get into. All you need is a glib tongue, a kitchen table and no job. There’s no licensing, no certification and damn few courses in how to do it better. Way too much training emphasis is on the moderation rather than analysis.

So we’ve got thousands of non-interpretative focus group moderators out there servicing tens of thousands of clients who don’t much care for interpretation perhaps because they’ve never experienced any other way to go about things. No wonder some people see focus groups as being flimsy research at best.

Shore up the techniques

But is it the form that should be faulted or the practices by which that form is being implemented? Are there ways to shore up focus group techniques so that they aren’t as misleading as people like Yahoo!’s Dunway have come to expect?

It has been my experience that focus group clients can mitigate the likelihood of being misled by taking a number of steps in developing a qualitative research approach. Here are three thoughts:

1. Participate actively in the research process.

Roll up your sleeves and work with your moderator in developing a real understanding of the consumers you expect to target. A briefing at the beginning of a project is never enough, no matter how thorough it is.

Although I understand the need for it, I regret the decline in back room viewers brought about by teleconferencing and the like. Don’t forget that the folks in the back room know much more about the topic than the moderator ever does. When I travel with clients I always find out something critical about the project that no one would ever think to tell me in a briefing. Listening to the conversation in the back room after a group (or at dinner later) almost always adds savvy to the analysis.

Listen, argue, suggest and explore all the way through the process. Good qualitative is an iterative process; the last group should always be different than the first.

2. Be patient with the interviewing process.

In order to parse out what is empirical and what is social truth you have to hear people talk about both. And then you have to confront them with the discrepancies.

I can’t tell you how many times clients have asked me to hurry through the “background” exploration with respondents. “We’ve heard that stuff a million times” is the usual argument. But while you may have heard that stuff before, you haven’t heard it from these particular respondents. And listening to them tell their story allows one to understand how that person’s empirical life strays from what they tell you when they are disposed to spouting social truth. And that, in turn, allows the moderator to confront people about those differences.

So, getting background information from people isn’t a warm-up exercise so much as a foray into empirical truths about the people to whom you are speaking.

It’s natural to want to get to the heart of the exercise of course, but it’s important to get a feel for who these people are first so that when you get there you have an empirical perspective from which to evaluate that individual’s point of view.

3. Encourage meaningful analysis.

Clients who provide moderators with the wherewithal (time, patience and funding) and who let it be known that they expect to follow the moderator’s analytic logic are likely to get reports that parse out what is social truth and what is behavioral truth. They are, accordingly, less likely to be misled.

Tell your moderator/analyst that:

  • You expect and need a report that goes well beyond summarizing what happened in the group - you require analyses that offer interpretations of what those things mean.

A good moderator should be an expert in translating consumerese into corporate language. “This is what people say and this is what you should take away from that” should be part of what a moderator gives you.

Don’t let some hoo-hah about subjectivity deter you. If you choose to believe what the consumer says literally, that’s your prerogative. But I suspect your moderator will be closer to objectivity than you.

  • You expect and need a report that makes recommendations about how to use this information.

A seasoned moderator has worked with lots of concepts similar in some ways to what you’ve got. He or she has internal norms that have been built over years of seeing his or her recommendations succeed or fail. He or she is going to be much better at predicting future behavior than people are themselves.

If the moderator’s recommendations are impractical, it’s a good sign that the concept isn’t that good anyway. If his or her recommendations are foolish it means you are operating with information he or she doesn’t have and needs to be better informed for the next study.

  • You need to follow the logic of what he or she has to say.

A good analyst tells you how they arrived at their translation of consumer-speak. He or she shouldn’t just give another point of view - he or she needs to build a ramp of understanding that you can follow to the same place. If the moderator doesn’t (or does so badly), challenge him or her on that.

  • You intend to read the report and talk it over in detail with him or her well before it is presented to management. Then make that happen.

Working with a moderator on a report forces him or her to codify his or her thoughts before the work is over and allows you to buy into his or her logic (or not) before it is set in concrete at a management presentation. Incidentally, research is most valuable on the day it is presented - it loses weight from that point on.

Subtle clues

One last thing: The entire concept of Blink, it seems to me, is based on people’s ability to make use of the subtle clues our brains attend to even when, at high levels of consciousness, we don’t recognize the process. But it takes place only when people are face-to-face with the subject matter, where they can read micro-expressions, smell the flop sweat on the liar in front of them, feel their own BS detector tingle. I am confident we are much less likely to blink effectively at arm’s length.

When it comes to collecting knowledge, I tend to at least partially agree with Marshall McLuhan’s contention that the medium is the message. As marketing research turns more and more to the efficiency of the Internet and other modern communication systems to generate reams of data in the name of reliability, the issue of validity must not be overlooked.

This is the worst of times to walk away from focus groups and other interactive, face-to-face explorations with the people you want to buy your concept. More than ever, industry needs the richness of new observations and serendipitous insights that take place when you sit across from people and talk. You need to be there when potential customers roll their eyes just before they tell you what a grand idea you’ve got.

As we move toward all-digital research, good focus group research is more important than ever. It’s essential so that you aren’t blindly assured that the beautiful bell-shaped curve spit out by the quant jock downstairs has you on a well-laid track downtown when your train’s on a shunt to the coal yard. It’s essential because it gives you and your team an opportunity to hear how real people, in a real social environment (more so than a “virtual environment” at least), think and talk about your ideas before you retool your production and distribution systems.

When you do this, retain a moderator who is an analyst as well as a facilitator and then work with him or her to parse out social and empirical realities. Use them to understand what’s good and not so good about your concept, how to optimize it in terms of those realities and what the likely response will be.