What do focus group facilities think of moderators?

Reflections in the mirror

Editor's note: Joel Reish is president of Next Level Research, Atlanta.

After 18 years in senior-level market research positions, I certainly have seen and heard a lot of opinions. Sure, many of these opinions were from respondents in surveys, focus groups, and other interviewing forums, but I have also heard a good number of opinions among market research practitioners. One category of opinions comes from focus group moderators about their experiences with focus group facilities - some good, many not.

I was struck, however, by the dearth of available opinions from facilities about moderators. Surely they have some things to say - thoughts and ideas that not only could add to our industry's general body of knowledge but also constructive criticism that moderators could use to improve their relationships with facilities and ultimately foster better qualitative research for their clients.

In investigating this topic I found that there has never been a quantitative survey of facilities' opinions of moderators. The two industry associations with memberships most targeted to these two parties, the Marketing Research Association (an association that many facilities belong to) and the Qualitative Research Consultants Association (an association of independent qualitative researchers), have sponsored some joint-session roundtables on the subject, which have been very revealing and constructive. But this survey is the first-ever of its kind, measuring quantitatively what focus facilities think of moderators - their strengths and shortcomings, what it's like to deal with them, and ideas for moderators to improve. The survey covered issues such as:

• Which areas of the relationship between facilities and moderators are sources of problems?
• How are moderators perceived?
• Which characteristics of a moderator might make a facility believe things will run more or less smoothly on a project?
• When should moderators arrive before and leave after groups?
• What is the expected impact of new technologies like videoconferencing and videostreaming?

Methodology

Having moderated focus groups for many years, including over five years with a company that owned focus facilities, I felt that I had a pretty good handle on most of the issues and potential problem areas between facilities and moderators. But to be sure that all relevant issues were explored in the quantitative survey, I first conducted one-on-one depth interviews with nine focus group facility owners and senior managers across the U.S. These respondents were from a mixture of big and small markets, from multi-market and single-market facility companies, and included some facilities I have worked with before, some I haven't worked with in a long time, some I have never worked with, and some picked at random. They are: Steve Schlesinger, Schlesinger Associates (nationwide); Greg Carter, Consumer Opinion Services (nationwide); Mimi Nichols, Nichols Research (San Francisco Bay area); Freddi Wayne, Fieldwork Dallas (Dallas); Christine Farber, INgather Research (Denver); Loren Wims, Marketing Horizons (St. Louis); Elaine Cravens, The Herron Group (Tampa); Frank Martin, Martin Focus Group Centres (throughout Virginia); Chris Cage, Indiana Research Service (Ft. Wayne).

After these interviews, the questionnaire was finalized for the quantitative survey, which was conducted online. To accomplish the various aspects of the survey, I partnered with several organizations. First, Quirk's graciously agreed to send out the e-mail invitations for the survey to all focus facilities in the U.S. and Canada that have an e-mail listing in the Quirk's Researcher SourceBook. We recognized that some multi-market facility companies publish one e-mail address for all of their facilities, and Quirk's Marketing Manager Dan Quirk was especially diligent at identifying each of these cases and making sure that multiple links to the survey were sent to such companies, with instructions to distribute the links internally, so that each facility would have an opportunity to participate in the survey.

For programming and Web hosting of the survey I partnered with Greenfield Online in Wilton, Conn., and for crosstabulation of the data I partnered with The Myers Group in Atlanta. I greatly appreciate all the help on this project from these organizations.

The e-mail invitations identified Next Level Research and myself as conducting the survey. The invitation was directed to the highest-ranking person in ownership or management at the facility and indicated that the results were to be presented at a major industry convention and would appear in an article in Quirk's Marketing Research Review. In addition to the initial invitations, two reminder waves were distributed during a two-week field window from September 13-27.

In all, invitations were sent to 752 valid e-mail addresses or individual facility listings in the U.S. and Canada, from which we received 219 completed interviews (a 29.1 percent response rate).

In the instructions to the survey itself, respondents were directed to think only about moderators, not end clients. In addition, respondents were instructed to recognize that the "moderator" may be more than one person from the facility's immediate client such as a moderator, field manager/director, project manager/director, account exec, etc., and that they should consider these collectively as "the moderator" for this survey. Finally, respondents were directed not to include opinions and experiences with in-house moderators or others in the respondent's own company.

Where are the problem areas facilities have with moderators?

First, we asked facilities to rate how often, in general, problems occur between facilities and moderators, using a 1-10 scale (1=never and 10=always). The average of their responses is an encouragingly low 2.9, with only 3 percent of respondents giving a high score in the 8-10 range. So, overall, these facility owners and managers are saying that problems do not occur very often between facilities and moderators. But delving into specific potential problem areas reveals another layer to the story.

The heart of the survey broke down for respondents the relationship between a focus facility and moderator into several phases:

• bid phase;
• recruiting phase;
• logistics and set-up phase;
• day of the groups;
• after the day of the groups;
• throughout the project.

Within these various phases, a total of 41 potential problem areas between facilities and moderators were identified, and respondents were asked to indicate which of these were actual problems for their facilities. A relatively consistent number of respondents identified at least one problem area for each phase, from 76 percent to 92 percent (Figure 1).

Of the 41 potential problem areas, seven were identified by more than half of all respondents as actual problems for their facilities in dealing with moderators (Figure 2). What is most interesting about these findings is how they are concentrated in the bid phase and recruiting phase (six of these top seven items) - this is where facilities are saying that the most common problems occur. And of course the seventh problem area, the timeframe to pay invoices, occurs in the phase after the groups.

Among those who indicated that each issue is an actual problem, that issue was further explored to gauge the severity of the problem when it does occur. Respondents rated the items they have found to be problem areas on a 1-10 scale (1=small/minor problem and 10=big/major problem). Of these top problem areas, those that cause the most trouble when they do occur are the timeframe to pay invoices, the usefulness of client sample, and how realistic the respondents specs and quotas are.

Exploring further the timeframe in which moderators pay the facility's invoice, a separate question was asked of these facility owners and managers to determine what percentage of moderators pay their invoice within 30 days. The responses here illustrate why the payment timeframe is such a common problem for facilities: over one-quarter of respondents (27 percent) say that only 10 percent or fewer of moderators pay the facility invoice on time! In contrast, fewer than one-quarter (23 percent) say that over 50 percent of moderators pay the facility invoice within 30 days, including only 4 percent of respondents saying that 91-100 percent of moderators pay on time.

There are eight additional problem areas that represent the next tier of issues, identified by 40-50 percent of respondents (Figure 3). These less common but still pervasive problem areas are more spread across the recruiting, logistics/set-up, and day-of-the-groups phases.

When asked to convey in an open-end fashion their advice to moderators, these facility respondents tended to indicate that they feel the following ideas and themes would alleviate many of the problems that do occur:

• communicate/give information, details and expectations;
• approach the relationship as a partnership - work together as a team;
• treat the staff with respect.

How do facilities perceive moderators?

The survey also explored the perceptions these facility owners and managers have of moderators. A total of 25 words or phrases were presented to respondents who were asked to rate, on a 1-10 scale, the degree to which moderators in general match the description (1=never and 10=always). These descriptors can be grouped into positive perceptions, negative perceptions, and other perceptions that are not necessarily negative or positive.

The results are a mixture of good news and bad news for moderators about the way they are perceived by focus facilities. On one hand, most of the positive images score more highly than the negative images. And further, the scores on the negative images all averaged to below the midpoint of the scale.

On the other hand, though, the positive images averaged mostly in the 5s and 6s on the 1-10 scale, and the negative images were not as low as I personally as a moderator would like to see them, averaging mostly in the 4s to mid-3s. (Figures 4-6).

So, facilities do not see moderators as strongly possessing negative traits, but they also do not see the positive traits in moderators as strongly as would be desirable.

What characteristics of moderators influence the expectations of facilities?

The perceptions of particular moderator characteristics were also explored in the survey. Respondents were presented with 20 different characteristics that might be held by a moderator and were asked, if this were all they knew about the moderator, would they expect that working on a project with that moderator would run more smoothly, less smoothly, or would that characteristic have no impact compared to other moderators in general. These characteristics include membership in various industry associations, size and kind of organization that the moderator works for, gender, age, and geographic location. And by subtracting the percentage who indicate a characteristic would lead them to expect a project would run less smoothly from the percentage who indicate they would expect it would run more smoothly, a "net impact" score can be generated for each characteristic.

The results are fascinating (Figure 7). While a majority of respondents said many of the characteristics would have no impact on their expectations of a moderator, the extent to which the others do say their expectations would be affected is revealing. The characteristics with the strongest positive impact by far are when the facility speaks directly with the moderator as the main contact during the bidding, recruiting and set-up phases of the project. These results are not at all surprising, given the already-shown feedback about the importance of communication and about how the most common problem areas tend to occur in the bid and recruiting phases.

Beyond that, respondents tend to have the most positive images of a moderator who is "one of their own" - namely, a moderator who is a member of the Marketing Research Association, a moderator who is local, and a moderator who works at another focus facility. Also having a large positive impact is a moderator who works for a small or medium-sized market research firm. In addition, membership in the Qualitative Research Consultants Association has a strong net positive impact.

Some facility owners and managers perceive moderators differently by age and by gender. While the majority of respondents say these characteristics have no influence on their expectations of a moderator, on balance more feel that a moderator being older than most has a net positive impact and being a moderator younger than most has a net negative impact. Being a male moderator has a slight positive effect and being a female moderator has a slight negative effect.

Several characteristics of a moderator have a noticeable net negative impact among facilities. On balance, facilities express a net negative impact for a moderator being from an advertising agency or marketing consultancy, a moderator being from New York, and when the facility speaks to a field/project manager leading up to the groups and does not speak to the moderator until the day of the groups.

Why ask some of these "politically incorrect" questions? Certain of these characteristics were tested in the survey because they were raised in the initial one-on-one depth interviews or reflect biases and stereotypes I have heard over the years. And for all of these issues, the intent of the research is simply to test possible preconceived notions that facilities might have of general moderator characteristics, certainly not to identify who is supposed to be a "good" or a "bad" moderator simply by the extent to which they match the profile of preferred characteristics.

Time to arrive, time to go

One set of complaints I have heard from facilities over the years involves how soon before the start of groups some moderators and clients arrive (sometimes too early and sometimes too close to the start of groups) and also how long after the end of groups some moderators and clients stay. But what do facilities view as reasonable arrival and departure windows?

The majority of facility respondents (68 percent) indicated that moderators and clients should arrive at the facility 60-90 minutes before the start of groups, without prior special arrangements. On the back end, 50 percent of all respondents say that up to 60 minutes after the scheduled end of the last group is the appropriate time window, and only 10 percent say that staying longer than 60 minutes is acceptable without prior special arrangements.

Videoconferencing and videostreaming technologies

Respondents were also asked if they feel that new technologies such as videoconferencing and videostreaming will make things better for facilities but not for moderators, better for moderators but not for facilities, better for both facilities and moderators, or not better for either. In general, most facility respondents (56 percent) say that these new technologies will make things better for both moderators and facilities.

But when we break out the responses of facilities that currently have videoconferencing and/or videostreaming capabilities from those who do not (based on self-reported data from an earlier question in this survey), the differences are striking. Among those facilities with such technologies, fully 76 percent feel that they will make things better for both facilities and moderators, and only 6 percent feel that these technologies will make things better for moderators but not for facilities. In contrast, only 35 percent of respondents without such technologies feel that they will make things better for both facilities and moderators, and fully 20 percent feel that these technologies will make things better for moderators but not for facilities (Figure 8).

Problem areas

On balance, facilities are saying that things generally go pretty smoothly with moderators, although they identified many problem areas. The most common problem areas tend to occur up front, in the bid and recruiting phase. In terms of specific descriptors, moderators are not seen by these facilities as strongly possessing negative traits, but they also do not see the positive traits in moderators as strongly as would be desirable. The factor seen as most positively impacting facility expectations is direct communication with the actual moderator in the early phases of the project. And facilities overall feel that communication and partnering are keys to a successful relationship between facilities and moderators. Do these findings mean that facilities are right and justified on all points? Not necessarily. Right or wrong, this is how facilities see moderators and this is the feedback they give. So, every moderator will better serve his or her clients' interests by heeding this advice, which will certainly foster a better relationship from the facilities' standpoint with moderators:

• Communicate...often and throughout the project.
• Foster a partnering relationship.
• Express understanding and empathy with facilities and the issues they face.
• Trust facilities and treat them with respect.
• Work towards solutions to problems.
• Let them know you belong to the MRA, the QRCA, and other industry associations.
• When using a client sample, expect problems, give extra time, and be prepared to pay more.
• Pay facilities in a timely manner.

And what about some of the more negative findings about characteristics that a moderator cannot change? For example, suppose you are a young female moderator from New York who works for an ad agency - do these research findings mean that you are doomed to a poor working relationship with focus facilities? Not at all. What the research does suggest is that such a person might face some stereotyping, but of course no individual is a stereotype, and every moderator can find ideas and suggestions from this research to help overcome whatever preconceived notions may exist about their circumstances and foster the most positive working relationship possible.

Goal is to succeed

Many facilities in their open-end comments indicated that their goal is for every client project to succeed and to make the moderator shine in front of his or her client; they just ask to be given an appropriate opportunity to do so. Here are some selected verbatim comments from these facilities:

"...Most miracles only require 48 hours advance notice..."

"...We're all in this together..."

"...Remember that we are partners with one goal, pleasing your clients..."

"...We are not the enemy..."

"...You have only rented the facility, you should not act like you own it..."

"...Please treat us like equals...because some of us are!"

"...We are not a bank..."

"...Take a deep breath - relax - trust your facilities to do the right thing..."