A focus group experiment tests remote moderating by telephone and video

By Sharon Livingston, Peter DePaulo, Joel Raphael | July 1, 2002

Reading time: 9 minutes

Abstract

The online market research community is flourishing. This article discusses the growth of online research, highlighting the importance of privacy and the challenge of security.

Research Topics:: Focus Group-Moderating | Focus Group-Videoconference | Qualitative Research
Content Type: Research Methodology

Share Print

On the line or on the screen

Editor's note: Pete DePaulo is an independent research consultant based in suburban Philadelphia. Sharon Livingston is president of Executive Solutions, Inc., a Syosset, N.Y., marketing research firm. Joel Raphael is president of ViewPower, Inc., a New York research consultancy.

Can productive focus groups be conducted remotely, that is, by teleconference or videoconference? Do we lose too much if the moderator is not physically present in the traditional way, sitting face-to-face with respondents in the same room?

Conversely, might there actually be some research advantages to such arrangements? Or, is remote moderating merely an expedient for cutting the cost and risk of travel?

These issues are timely. In the current tight economic climate and the aftermath of September 11, moderators and their clients are particularly interested in options for conducting interviews remotely. Also, improvements in technology and the Internet are enhancing the quality and adaptability of videoconferencing, videostreaming to PCs and other forms of remote communication for focus group research.

To gain some insight into the quality of remote-moderating options, Executive Solutions, Inc., recently sponsored a qualitative experiment at its New York regional facility, The Looking Glass. What made this experiment unique was that the same group of respondents discussed the same general topic under three different modes of interaction: by telephone, remote video, and traditional face-to-face. The study was co-sponsored by FocusVision, Inc., which provided the remote video capability.

The experiment

Actually, much of this exercise was rather typical for focus groups. The discussion subject was a local shopping center, the Walt Whitman Mall in Long Island. The respondents were six female patrons of Whitman and other local malls. The group included young adults as well as middle-aged women, and focus group veterans as well as novice ("virgin") respondents. Specific questions included shopping habits, perceptions of the Whitman center, reasons for preferring one mall versus another, and suggestions for improving Whitman. Moderators were highly experienced practitioners who have been in the qualitative research profession for decades.

What was not typical was the division of the 90-minute discussion into three separate sets of questions, administered in different modalities and by different moderators.

1. Telephone focus group phase. After arriving, the participants were seated at separate telephone cubicles. They were asked to imagine that they were at home and were to dial into a conference call. The call was routed through the telecommunications equipment of Market Navigation, a research company that provides telephone focus group services. George Silverman, president of Market Navigation, conducted this phase of the focus group. While moderating, Silverman was able to watch his laptop computer screen, which identified the respondent talking at the moment. (This computer-screen ID is a component of Silverman's system.) During this phase, he actually was in another room at The Looking Glass. Of course, he could have been anywhere in the world with telephone lines.

2. Remote video phase. After completing the telephone phase, the respondents were brought together into a room where they simultaneously watched a TV monitor showing the live "talking head" moderator, Sharon Livingston of Executive Solutions. Like Silverman, Livingston actually was in another location on the premises but could have been anywhere with suitable video equipment.

3. Traditional face-to-face. For this last half hour, respondents were brought into one of the Looking Glass's focus group rooms and seated around the conference table for a traditional face-to-face discussion. Joel Raphael of ViewPower was the moderator for this phase. After completing the discussion of the shopping mall, Raphael asked the respondents to share their feelings about the three modalities, including how comfortable they were with each way of participating.

Although each moderator asked a different set of questions, all three concentrated on the same subject - the shopping mall. The second and third moderators built on the prior discussions, just as the later questions in a typical qualitative interview relate to the earlier ones.

Observers

In order to explore the reactions of research professionals to this experiment, Executive Solutions invited a number of observers - mostly other qualitative consultants from the mid-Atlantic region - to the back room. They watched all three moderators in action, and were also able to see participants in the video and face-to-face phases. Later, these on-site observers participated in debrief discussions with Silverman, Livingston, and Raphael, regarding what worked and the relative advantages of each modality.

In addition to observers in the real back room, there also were remote observers around the country. They were able to watch the proceedings thanks to an Internet broadcast arranged by FocusVision's videoconferencing service. A number of these observers instant-messaged or called in questions during the proceedings and debriefs.

Qualitative results

The big picture is that all three modalities worked well in stimulating respondents to share their thoughts and feelings. Respondents engaged in productive conversations throughout the three phases. The professional moderators who conducted the experiment came away confident that telephone and talking head focus groups can generate genuine qualitative insights in the same league as face-to-face groups.

Telephone results. Observers, most of whom had never conducted a telephone focus group, enjoyed the smoothness of the phone phase. Respondents opened up immediately after Silverman's introduction, and verbally confirmed their high level of comfort with the telephone session when Raphael asked them about it at the end of the third phase. As Silverman pointed out, we should hardly be surprised at respondents' ease right from the get-go, as people use telephones in all kinds of situations in their daily lives.

During the telephone session, respondents rarely talked over each other. They politely took turns, unlike the case with face-to-face respondents, who have to be reminded not to talk all at once. In fact, when the face-to-face session started, respondents almost immediately began talking in multiple simultaneous conversations until Raphael brought them under control. This was powerful evidence that a telephone focus group has the advantage of avoiding the frustrating, garbled recording you get when respondents talk simultaneously.

In audiotapes from traditional face-to-face groups, it is usually hard to tell who is talking, except for the respondents with the most distinctive voices. This is not a problem in Silverman's methodology for telephone groups, because he asks them to mention their first names whenever they talk. Because the respondents do not know that Silverman is watching a screen identifying the talkers, they oblige. This request would seem strange in a face-to-face group, where respondents can see each other's name tags. But in the telephone groups, giving their name when they talk identifies them to each other.

Livingston offered a motivational reason why it is good idea for respondents to identify themselves when they speak: by doing so, they take more responsibility for what they say.

Of course, telephone focus groups have a potential downside, although it is easy to overstate its importance: the moderator does not see the respondents. Researchers who have not conducted or heard telephone focus groups are understandably concerned about the loss of the visual cues such as facial expressions.

Interestingly, there is evidence from academic studies in social psychology suggesting that in judging a person's sincerity, visual cues add little to what is evident from hearing the person talk. In fact, when the observers at The Looking Glass were listening to the telephone focus group during the experiment, they noticed that respondents' feelings were quite evident in their voices.

Neither our experiment nor the academic studies suggest that visual cues are useless in qualitative research. During a telephone group, the moderator cannot watch how other respondents are nonverbally reacting to the respondent who is talking. Nevertheless, it seems clear to us that telephone focus groups are an underutilized and under-appreciated methodology.

For Silverman and others who have conducted telephone focus groups, these findings were no surprise. We emphasize them here because the advantages and quality of telephone focus groups still are not widely known. Perhaps this lack of awareness explains why Internet focus groups are typically compared only with traditional face-to-face groups, rather than also with telephone groups.

Talking-head results. In this phase, Livingston, the moderator, did have access to each respondent's visual cues (and likewise, respondents could see her), through a video monitor.

As they began the session, respondents continued to engage in the productive conversation that started in the telephone phase. When they later reflected on their "talking head" experience in the debrief, they admitted to feeling a little strange at the start of the remote video session - presumably because they aren't used to interacting with a person on a TV set. However, the respondents quickly got used to this, as was evident to observers watching the respondents on closed-circuit TV. They also paid close attention to the monitor, watching the "show host" with focused interest.

One of the advantages of the talking-head setup is the ability to observe respondents' body language. Although Livingston was "remote," she was able to observe and zoom in on individuals' facial expressions and other non-verbal behavior - which would not be observable, of course, on the telephone. She was able to call on a particular respondent when body language silently signaled that the person might have something to say. When one shifted in her chair as another was talking, Livingston called on her next to check her response to the comment just made. This reading of the non-verbals flowed as naturally as it would in a conventional in-person setting. In fact, the remote camera seemed to make reading the group more efficient for the moderator, since participants were not able to tell exactly who Livingston was looking at moment to moment. This scanning of the group for non-verbals would have to be done more discreetly in-person.

There were some other fascinating elements unique to the video conferencing setup that appeared to make eliciting information potentially even more effective than a conventional setup.

Video leadership is more intriguing and exciting to the group members. Seeing the moderator on TV may impart an almost celebrity status to the facilitator as it does to a newscaster or host on a regular TV show. This effect seemed to contribute to heightened interest and energetic participation during the group session. The effect of the TV was confirmed when Livingston later met the respondents in person, and they behaved in an almost starstruck manner. Suddenly you're no longer a Regular Joe (or Joanne), but are someone special, it appears.
People enjoyed the unique experience of talking to the television and having it talk back. The interactivity factor made the sessions more stimulating for participants. As was the case in the telephone group, respondents mostly took turns relating to the TV host and each other, rather than engaging in simultaneous conversations.

Enhance communication

In her questioning, the moderator applied a few projective techniques. The guided imagery worked effectively, as it does in a conventional focus group, with responses that were full and rich. Thus, our experiment suggests that remote moderation allows the leader to draw out psychologically-motivated material in addition to responses to direct questioning.

Remote moderating can also enhance communication with the back room, which no longer needs to be situated in the dark behind a two-way mirror. If desired, client observers can be in the same room with the moderator, just out of camera or telephone range, and thus more conveniently and effectively communicate with the moderator.

So, what's the bottom line on remote moderating? Both telephone and TV work and work well. In some cases they may work even better than conventional groups - and they have the potential to save clients money.

What can we conclude?

Our qualitative experiment was, of course, not conclusive in the quantitative sense. It was not a scientifically controlled experiment with repeated measures in counterbalanced sequences of modalities, nor was it intended to be. Nevertheless, for the observers and moderators, it was compelling. This is analogous to clients sometimes coming away from a focus group with the intuitive confidence to make a marketing decision even though they know full well that qualitative results are not projectable statistically. Likewise, the moderators who observed the process came away with the firm impression that remote moderating by telephone and remote video really works. They now are eager to do remote interviewing themselves.

We believe that remote moderating should be undertaken with the positive attitude that the client can gain a great deal of rich insights. It is time to move beyond the presumption that any focus group without the moderator physically present in the same room with respondents is merely a weak, "better than nothing" substitute to be used only when absolutely necessary. Indeed, when it is important to hear the full range of views of the target market, at least some of the groups should be by telephone in order to hear from the type of respondents who cannot or will not come to a focus group facility.

Similarly, remote video moderating should be viewed positively as a means to make the best use of moderators' time, enabling them to do more groups and less traveling. If, for example, a small group of highly valued respondents (e.g., experts or executives) is an ocean away, remote video monitoring can allow clients to "see" this crucial market segment talking about the product without the delays, costs, and risks of world travel by the moderator.