Keeping dead wood out of focus groups - a screening question that improves the odds

Editor’s note: Neil Kalt is president of Newtown, Conn.-based research firm Neil Kalt & Associates.

As anyone who’s moderated or observed focus groups knows, all respondents are not created equal. At one end of the continuum are people who are intelligent, imaginative, insightful, involved, and clearly able to express their thoughts and feelings. At the other end of the continuum are people who are neither intelligent nor insightful, who have considerable difficulty expressing their thoughts and feelings and who, if present in sufficient numbers, weigh on a group like a ball and chain. If you’re lucky, most of the respondents in your groups will be drawn from the favorable half of the continuum. The problem is, it takes more luck than most people have to continually defy the laws of probability. Which is why more than a few moderators try to build a litmus test into their screening questionnaires - often an open-ended question that’s designed to separate the dead wood from the live. For example, “If you could speak with anyone who ever lived, who would you choose to talk to, and what would you talk about?” Or, “Think about a movie or a book that you really liked, then tell me what made it special.”

Shortcomings of open-ended questions

Does this type of question work? In the absence of data, we don’t really know. However, there are several reasons for feeling that if it does work, it’s not nearly as often as we’d like:

• At least some of these questions may be fairly easy for sizable numbers of people from both halves of the continuum to answer. If they are, their answers can’t tell us who is likely to contribute to the discussion and who is not.
• Everyone recognizes a highly articulate, insightful answer when they hear it. However, opinions begin to differ as one moves away from the favorable end of the continuum and toward its center. When is an answer not sufficiently articulate and insightful? Unfortunately, it depends on the perceptions and expectations of the recruiter who’s doing the listening.
• This type of question may have a good chance of being taken seriously when the screening questionnaire is short and the incidence of eligible respondents is high. However, as the length of the screener increases and/or the incidence declines, it seems increasingly likely that recruiters will “stretch” their definitions of “sufficiently articulate and insightful,” perhaps enough to include all but a handful of people. Pressed by the constraints of time and money, recruiters want to find people who meet the project’s “real” eligibility requirements, get them to agree to participate, then move on to the next job.

Separating the taciturn from the talkative

Uncomfortable with using open-ended questions in an attempt to weed out the dead wood, some moderators prefer a closed-end question that’s designed to separate the shy and uncommunicative from the outgoing and talkative. Although what constitutes an acceptable answer is no longer left to the discretion of recruiters, I suspect that this type of question is no more effective than its open-ended brethren. There are at least two reasons why. The first is social desirability. Most people prefer to see themselves in a socially favorable light. Accordingly, they’ll tend to choose socially desirable answers, even if these answers are less than true. The second problem with questions of this kind is the increased risk of recruiting people who love to talk but have little or nothing of value to say. Indeed, because they take time off the clock, these people can be more detrimental to a group than respondents who are inarticulate.

A new paradigm

What’s to be done? To heighten the probability that it will work, a question that’s designed to separate the wheat from the chaff should meet two criteria: it should have answers that are unequivocally right or wrong, thereby taking the discretion to decide out of recruiters’ hands; and it should be answered honestly by respondents.

• To have answers that are unequivocally right or wrong, the question must be closed-ended.
• To be answered honestly, the effects of social desirability must be marginalized. One way to accomplish this is to make the attributes that lead to termination as appealing as the attributes that lead to an invitation to participate.

Let’s look at some studies in which this paradigm was used.

Example 1
In a study that sought to shed light on the imagery of a leading brand of women’s shampoo, we wanted to recruit women who were open, intuitive and imaginative. Not surprisingly, we felt that women who were measured, rational and matter-of-fact would not fit the mold. In an attempt to discriminate between these two types, we presented each woman who was screened with five pairs of attributes. In every pair, both attributes were considered to be favorable. Taken together, the attributes that are asterisked below described the kind of woman we wanted to recruit. The other attributes described the kind of woman we did not want to recruit.

The women were asked to indicate “which attribute in each pair comes closer to describing you.” To qualify, a respondent had to select at least three of the five attributes that had an asterisk (respondents were read one pair at a time and asked to make a choice):

Example 2
Let’s look at another, more elaborate example. In a number of studies that called for creative people — people who were open to as well as capable of generating new ideas - 12 pairs of attributes were used. In nine of these pairs, one of the attributes was chosen because it seemed to describe creative, imaginative, insightful people. The other attribute in each of these pairs was chosen because it appeared to meet two criteria: it was appealing and was unrelated to creativity. To further mask this question’s intent, three pairs of favorable attributes in which neither attribute was related to creativity were added to the mix. In order to qualify, respondents had to say that at least six of the nine creativity-related attributes came closer to describing them. Not wanting to force respondents to choose attributes that weren’t reasonably accurate descriptors, they were given a third choice: “Neither.” Some of the pairs that comprised this question are as follows (qualifying attributes are asterisked:

Two findings suggest that this question is working. In every one of the projects in which it was used (all of which employed elaborate screeners), a fair number of people terminated on this question, implying that recruiters treated it as they would any other screening question. Moreover, many of the people that were recruited did appear to be insightful and imaginative.

In addition to being closed-ended and making every answer socially desirable, one reason why this question may work is that it gets rid of the dead wood indirectly - that is, it doesn’t try to identify people who have little or nothing to contribute. Instead, it seeks to identify and recruit people at the other end of the continuum, people who make focus groups glitter. The people who are dead wood are terminated along with everyone else who falls short on this question.

Benefits

It seems fair to conclude that the benefits of this approach are considerable:

• Deciding who qualifies isn’t left to the discretion of recruiters.
• This approach is flexible; that is, the attributes can be tailored to the requirements of the project.
• By presenting respondents with pairs of favorable attributes, this approach lessens, and may even minimize or negate, the effects of social desirability. In so doing, it increases the likelihood that respondents will answer honestly.
• Because the tasks that are presented to the group dovetail with the characteristics of the people who have been recruited, low levels of involvement and expressiveness are much less likely to occur.

Despite its benefits, this approach is not foolproof. A lot depends on the attributes that you choose. For example, to what extent do these attributes capture the psychological profile of the people you want to recruit? And to what extent are the attributes that they’re paired with as socially desirable? Prudence dictates that you re-screen at the door, and send home any dead wood that managed to slip through the net.