Editor's note: Jim Schwartz, Ph.D., is market research manager at Newsweek magazine. He gratefully acknowledges the editorial comments of Gene Waggaman and John Wittenbraker during the writing of this article.
Critical to the success of every focus group is the recruitment of properly screened respondents. Intentional mix-recruitment is dishonest and unethical. If it does occur and is revealed, all data received from improperly screened respondents must be considered invalid. In addition, the effect of these respondents on other group participants must be considered, since their comments may undermine the validity of the entire findings. A case study of an improper recruitment follows.
Recruiter: Hello, Dr. Schwartz. I'm Alice from ZYX Field Service. We are conducting a focus group with frequent travelers like you and would like to offer you an honorarium of $100 to participate. Would you like to participate?
Questions: How did she know my title? How does she know I'm a frequent traveler? From what list did she get my name?
Comment: Not mentioning her last name is, I believe, unprofessional. The amount she offered is quite nice. If it is convenient for me to participate, why shouldn't I?
Jim Schwartz: Perhaps. First tell me when and where it will be.
Rec: There are two choices of one and a half hour each - at either 6:15 p.m. or 7:45 p.m., on Tuesday, October 9 at (location).
JS: O.K. I'll attend the 7:45 p.m. session. Thank you for calling.
Rec: No, no, wait. Before you go I have to ask you a few questions. First, are you or is anyone in your household employed by a market research company? A market research department? Hotel? Motel? Airline? Advertising agency? Car rental company?
Comment: This is indeed a "strange" recruitment interview. She has already given me the information for attending and offered me $100. Now she wants to ask me some questions? If I tell her that I work for a market research department she will disqualify me. Actually, I feel that this particular screening criterion is often unnecessary. To see what tack she will take next, I will say "no" to all of the above. I can always tell her at the end of the interview I've changed my mind.
JS: No.
Rec: Have you participated in any focus group in the past 12 months?
JS: No.
Comment: I have moderated and observed groups during my career, but have never been recruited to be a participant in one.
Rec: Please tell me which of the following frequent visitor programs you are an active member of. Hotel A?
JS: Yes.
Rec: Hotel B? JS: Yes.
Rec: Hotel C? JS: Yes.
Rec: Hotel D? JS: Yes.
Rec: Hotel E? JS: No.
Rec: Hotel F? JS: Yes.
Rec: Please tell me how many days in the last 12 months you have stayed at hotels in each of the following chains. Hotel A ?
JS: Three.
Rec: Hotel B? JS: Six.
Rec: Hotel C? JS: None.
Rec: Hotel D? JS: Two.
Rec: Hotel E? JS: None.
Rec: Hotel F? JS: None.
Rec: Try and think about Hotel F again. Can you stretch your answer to "at least twice" in the last twelve months? Try and stretch.
Comment: This is quite remarkable. She is now asking me to lie. She must be desperate to recruit people. All right, I'll accommodate her.
JS: If you want me to say "two," I'll say "two."
Rec: Good. Now, please tell me how many different hotels in each of these chains you have stayed at during the last 12 months. Hotel A?
JS: One.
Rec: Hotel B?
JS: One.
Rec: Hotel C?
JS: None.
Comment: This query is unnecessary, because I just told her I have not stayed at a Hotel C even one day in the last 12 months. The script should have called for a skip here. I wonder if it did.
Rec: Hotel D?
JS: One.
Rec: Hotel E?
JS: None.
Comment: If she has a well designed screener with appropriate skip patterns, she is not following it properly.
Rec: Hotel F?
Comment: I now know this chain is particularly important to this project. I actually have not stayed at any of their properties, but I want to find out what she wants here.
JS: Is "none" OK?
Rec: No. Try and stretch. Try and think if you have stayed in at least two different Hotel F hotels in the past 12 months.
JS: OK. If that is what you want (I'll lie for you), two.
Rec: If you had an eleven-day trip to schedule, please tell me how many days you would stay at each of these chains.
JS: (Let me see if she will spill her beans here.) Does this question matter?
Rec: No, it doesn't. Can I help you with this ? Let's mark down six for Hotel B, three for Hotel E, two for Hotel A, and zero for the others.
Comment: It is fascinating to see the irrelevancy of reality to her and her ease in supplying the answers for me.
Rec: OK, that completes the interview. You will get a card in the mail confirming the date, time and with directions. Let me confirm your address, etc.
Observations
Research is a service industry. It involves a high degree of trust among its varied segments. The field services are the pillar responsible for recruiting and interviewing. If they cannot be relied upon to execute their responsibilities properly, the entire research industry is endangered.
Recruitment is the foundation of a good focus group. Every focus group recruiter should be provided with a clear, professionally crafted recruitment form and instructed in its proper use. As I later learned, the recruiter who called me had such a form. Unfortunately she did not follow it. She did not just slightly deviate from the script; this was a case of gross misconduct on the recruiter's part. If she had felt the script was deficient in some way, she, through her supervisor, should have worked with the moderator who arranged for the focus groups to be conducted at her facility to improve it.
She made a cardinal error in offering me a significant honorarium prior to my having met the screening criteria. Having been offered such an incentive, a typical respondent, being only human and perhaps too easily enticed by money, might be tempted to not let the opportunity pass quickly. However, this recruiter compounded the error not only by encouraging me to lie, but even feeding me the "correct" answers.
Though perhaps tired or exasperated by the difficulties of locating properly qualified respondents, she, by her actions, threatened the validity of the entire project. False or biased recruiting procedures inevitably lead to group participants who provide defective information. The long and short of it is that a script must be properly followed, and the incentive for participation must never be offered before other screening criteria have been correctly met.
Outcome
Did I participate in the focus group? Did I accept the honorarium and give the sponsor biased information? No. After some reflection and discussion with colleagues I decided the best way to extricate myself from this project, so I would not further compound possible error, was to try to reach the study's sponsor. I would have preferred speaking to the moderator, but had no idea of his or her identity. I suspected the sponsor was Hotel F, with the most likely alternative being that chain's leading competitor. Calling Hotel F's corporate headquarters, I spoke to the market research director and asked her to confirm that her company was sponsoring the focus group to which I had been invited. Although shocked that I called, she said "Yes." I then described the fashion in which I had been interviewed. She said that, as a result of my experience, the entire project was endangered, and might have to be canceled. My invitation to attend the group was, of course, withdrawn.
I emphasized to this researcher that my purpose in calling was not to get the recruiter fired. It was simply to have her, as the sponsor, realize the faulty procedure used for my recruitment, and then decide how best to proceed. Whatever steps she took, I suggested, should include additional education for this particular recruiter.
Suggestions for avoiding mis-recruitment
Recruiting proper respondents is often a difficult and time consuming task. Here are several suggestions which may help reduce the likelihood of mis-recruitment.
First, and most obviously, a proper recruitment form must be prepared. As with every questionnaire, it should be as simple as possible. No "nice to know" questions should be included. To minimize the interview's length, questions requiring skips should be clearly noted.
Second, the screening criteria that determine a participant's eligibility should be kept to a minimum. For example, in this particular study, was it truly necessary that employment at advertising agencies, market research companies/departments, airlines or car rental companies, of either the person phoned or anyone in the household, serve as a criterion for rejecting a possible participant? I understand a client's interest in minimizing the possibility that a direct competitor, in this case an employee of a hotel or motel chain, benefit from proprietary research. However, I believe the longer list of exclusions is unnecessary, and denies a sponsor the opportunity to receive data from many talented people.
This opinion enables me to believe that my response to the question about employment in a market research department, while inaccurate, was not unethical. (Some researchers, of course, will disagree with me.)
Third, the pressures recruiters face must be recognized by clients and reduced as much as possible. For example, recruiters should be provided with adequate time to recruit. About two weeks may be optimum.
Fourth, quality control checks should be built into the recruitment process.
For example, if a call to confirm participation is made the day before a focus group, an interviewer other than the original one should be used. And the confirmation form could contain a couple of validation questions. Alternatively, respondents could be rescreened by another person when they arrive at the focus group facility.
Last, flexibility in the research process is necessary. For example, little is gained by being rigid about the number of participants recruited if unusual difficulties are being encountered. A group of five completely qualified respondents providing accurate information is far more valuable to a project sponsor than a group of ten respondents that includes some who are marginally or improperly qualified. Quality, not quantity, is the operative principal.