Guidelines for screening and re-screening

Abstract

The Marketing Research Association and Qualitative Research Consultants Association’s Joint Committee has been working on the issues of cheaters and repeaters during 1999. This article reports on the most recent recommendations of the Joint Committee’s Screener/Re-screener Task Force.

You get what you ask for!

Editor's note: Merrill Shugoll is president of Shugoll Research, Bethesda, Md., and Screener/Re-screener Task Force chair of the MRA/QRCA Joint Committee. Nancy Kolkebeck is vice president of JRA, J. Reckner Associates, Inc., a Montgomeryville, Pa., research firm.

In 1996 the Marketing Research Association (MRA) and the Qualitative Research Consultants Association (QRCA) established the MRA/QRCA Joint Committee to address issues of mutual concern. The outcome of their first effort was published in the December 1998 issue of Quirk's as "Guidelines for the future: recommended best practices for qualitative database development and usage."

During 1999, the Joint Committee has been working on the issue of "cheaters" and "repeaters." (Cheaters are potential respondents who lie about demographic or other criteria in order to participate in a focus group. Repeaters are those who have participated in more than one focus group in a six-month period and who fail to disclose this information during screening.) Several task forces have been established to investigate tools to address the industry's concern regarding professional respondents. This article reports on the most recent recommendations of the Joint Committee's Screener/Re-screener Task Force. The authors wish to thank task force members Rebecca Day, Angela Lorinchak, Kevin Smith and Cyndi Tompkins for their dedication and hard work.

These guidelines are presented to help qualitative researchers improve the quality of screening and re-screening instruments. The objectives are to assure the recruitment of fully qualified participants who match the target population desired by our clients, and to reduce the incidence of cheating and repeating.

In order to minimize cheating and repeating, a three-phase screening process is recommended. First, initial recruitment should be accomplished by using a screening questionnaire. Second, re-screening should be done during the confirmation call by a monitor or supervisor not by the original recruiter. Finally, on site re-screening should be conducted when the respondent arrives at the facility. To that end, what follows are recommendations for screener and re-screener design.

General rules of thumb for screener design

Screeners are and will always be imperfect documents. A five-minute screening questionnaire cannot give us a complete understanding of respondents' attitudes and behaviors. And, we can never eliminate all mistakes that can occur during screening. However, we can limit mistakes by writing better screeners. In designing screeners, we need to follow the basic principles that also apply to questionnaire design. Some basic do's and don'ts are:

DO start with general questions and proceed to more specific questions.
DO write questions that use consumer language not client language.
DO ensure that all possible answer categories are provided if the question is close-ended. Remember to include "Other-Specify" and "Refused" categories.
DO tell respondents in the invitation section that they do not have to do anything to "prepare" for the session. One facility manager indicated that "trier rejecters" and "aware non-triers" sometimes go out to purchase products prior to the groups.
DO substitute open-ended questions for scales whenever possible and appropriate.

Open-ended questions are good because they:

- are more difficult to fake because there is no indication of what the right answers are;

- show what a respondent is thinking in his/her own words;

- provide additional opportunities to assess articulation.

DO NOT compound two questions into one.
DO NOT ask leading or biased questions.
DO NOT exceed 20 words per question whenever possible.
DO NOT, if possible, design a screener that exceeds seven minutes.
DO NOT ask questions that are personal/confidential unless they are essential for analytical purposes.
DO NOT terminate on a key question, if possible. Instead, THANK AND TERMINATE on an innocuous question. This will keep professional respondents from understanding what the screening criteria are.

Screener introductions

Introductions are important because they set the stage for whether the respondents will allow the recruiter to continue the screening process. However, if not properly written, introductions can bias respondents toward trying to qualify in order to participate and earn the incentive money. DO identify the recruiter's name and recruiting agency's name.

DO keep it very brief, but give enough detail to make the project sound interesting and enjoyable.
DO include the topic. If a client list or other cold sample is being used, the introduction also needs a purpose and how respondents will benefit from participating.
DO include the client's name, if permitted to do so and if it would increase cooperation.
DO tell respondents we are not selling anything, we are calling strictly for opinion research purposes.
DO NOT give the specifics: date, time, location, type of project (groups vs. IDIs), gratuity, etc. This information should be in the invitation section and only divulged after we have identified qualified respondents.

Product/service category/brand usage questions

Improperly designed usage questions lead to non-qualified respondents. Pay careful attention to how these questions are structured. Design usage questions with the following pointers in mind:

DO use open-ended questions when seeking "most often" or "top-of-mind" behaviors. (Example 1)
DO NOT ask open-ended questions to identify behaviors or products that occur infrequently. When looking for "non-users" it is critical to use an aided question.
DO NOT ask self-rating questions. If you must, use them in conjunction with an open-ended question. This enables a recruiting supervisor to assess how an individual's view of themselves compares to others in the group who have rated themselves similarly. (Example 2)
DO ask consumers to bring proof/evidence to the phone, if possible/applicable (e.g., product, bill, directory, etc.); if not possible, ask product/brand usage questions again at the end of the screener to confirm.
DO ask the facility, if possible, to buy all brands in the category so recruiters can tell by respondents' description if it is the correct brand.
DO be specific about usage frequency. (Example 3)

Demographic questions

The target market/audience for a product/service or communications program is often defined, in part, by demographics. Therefore, as qualitative researchers we must screen prospective respondents by such variables as gender, age, income, geography, marital status, family status, etc., to ensure that study participants are most representative of the target markets identified by clients.

DO use a transition sentence to introduce demographic questions to put respondents at ease and help them understand why certain questions are being asked (i.e., to ensure that a cross section of consumers are represented).
DO use multiple answer categories rather than "yes," "no" or other bipolar responses because it makes it more difficult for the respondent to guess what answer is being sought.
DO use exclusive (no overlapping) answer categories so that recruiters will not have trouble determining where to put respondents' answers. (Example 4)
DO use parallel answer categories in a question. (Example 5)
DO include one or two unnecessary answer categories, particularly for income and education, even if they represent a THANK & TERMINATE answer category so respondents do not have to lie/cheat to avoid falling into what is perceived by them as the least desirable answer category. An example would be adding "grade school" and "graduate degree" to the typical answer set for an education question.
DO update your answer categories to reflect current cultural trends.
DO ensure that your lower- and upper-income answer categories reflect the incomes in the selected geographic area.
DO specify geographic area mixes desired either by county, Zip codes or whatever geographic designation is pertinent.
DO ask about education if it is believed that income alone is not enough to identify the type of respondent that matches the client's target market.
DO clarify family status questions based on project needs/specifications. For example, if you are looking for households with children, be sure to differentiate between children under 18 living at home, children 18 or older living at home, children 18 or older living away from home.
DO consider asking a few psychographic questions or attitudinal questions using semantic differential scales or open-ended questions if demographics alone will not identify the target market for the product.
DO ask specific job title/occupation to classify respondents by white-collar, blue-collar, gray-collar, pink-collar jobs, rather than having respondents classify themselves.
DO NOT assume a facility will automatically screen for demographics that are not asked on the screener.
DO NOT apply rigid market statistics on demographic characteristics for focus group recruitment specifications in order to replicate the market (e.g., 20 percent of the market is 25 to 34 years of age, so two to three respondents must be 25 to 34 years of age).
DO NOT ask questions of respondents that pertain to others in their family. Only ask what is pertinent to the respondent, unless you are interviewing parents about their young children.
DO NOT ask sensitive demographic questions like age, income, and race at the beginning of a screener, if at all possible.

Occupation/security questions

Occupation or security questions are asked to determine if the respondent has an unrepresentative level of experience in the topic area that would place them in the position of being considered, by himself/herself or by the other respondents, to be an "authority" on the subject. Additionally, these questions are used to determine if any of the participants might pose a threat to client confidentiality if information discussed in the research is shared with others.

DO ask close-ended questions about occupation early in the screener and, as a double-check (if desired), open-ended questions near the end of the screener.
DO ask about employment of other members in the household, not just the spouse, if security is a significant concern.
DO NOT combine thoughts into a single question (e.g., "Are you or a member of your household . . .").
DO add other industries to the security screen if it is desirable to keep the topic of the research study confidential until the discussions are underway.
DO consider using a "Yes" or "No" format to require respondents to more carefully consider their responses to each question. (Example 6)

Past participation questions

Questions about past participation help us identify individuals who have become what are commonly classified as professional respondents. These individuals have done so many focus groups that they no longer represent the typical consumers we strive to include in our research.

DO ask the respondents the qualifying questions in a way that will not provide a clue to the answer that you are looking for. (Example 7)
DO be sure that the respondent understands that these questions relate to focus group studies that they have participated in anywhere - not just those conducted at the recruiting agency's facility. You may also want to exclude those who have participated in other types of research such as in-depths, taste tests, mall intercepts, and phone surveys.
DO ask the past participation question using lay language rather than market research jargon.
DO place the past participation question near the end of the screener. Provide a space for entering the respondent's last participation date, so that those respondents who have qualified on the remaining parts of the screener can be recalled if past participation requirements are relaxed. (Example 7)

Articulation, responsiveness, creativity questions and sensitive topics

The intent of articulation questions is not to screen out consumers whose opinions we value but to help increase the chance that those respondents who participate in our studies can clearly communicate their thoughts to others.

DO ask the articulation question as a part of the normal screening process, using open-ended questions that pertain to the screener or topic at hand: What are your job responsibilities? Tell me about the last time you used this type of product.
DO be specific in describing to recruiters what an "articulate" person means to you.

Responsiveness questions are used to determine if respondents are comfortable sharing ideas in a group setting and will be active participants.

DO assess the potential respondent's comfort in discussing ideas with others. (Example 8)

Creativity questions are used when it is important that those participating possess an aptitude for coming up with new ideas and/or reacting to ideas on the cutting edge. While many moderators like to use these questions, others believe that all consumers, when placed in a comfortable environment, have the necessary skills. Example 8 shows appropriate close-ended questions for those who are interested in identifying "creative" individuals. Other suggestions include:

- Use questions that are in sync with other topics addressed in the screener or everyday life rather than questions like "Tell me 10 things you could do with a paper clip."

- Over-recruit and personally interview each respondent upon arrival.

- Ask respondents to complete a homework assignment such as a collage related to the topic and select respondents on the work they have produced.

We are often asked to interview respondents about sensitive topics that may be considered extremely personal such as medical conditions or lifestyle topics. It is important to make respondents aware that they will be participating in a discussion with others who have similar experiences/lifestyles and/or that they will be asked to share their opinions on a specific topic.

DO order the qualifying questions so that respondents who may find the line of questioning offensive can be terminated before more sensitive questions are asked.
DO inform respondents, if appropriate, that they will be participating in a group discussion with other respondents who may share similar lifestyle, religious or political beliefs.

Re-screeners

The purpose of a re-screener is to assure that respondents qualify for the project and to determine which of the respondents who show up are best qualified to participate. To prepare a re-screener that will meet these goals and the goal of screening out cheaters and repeaters:

DO make sure questions on re-screeners are worded exactly as they were worded on the screener. However, response sections should be open-ended.
DO ask only key qualifying questions.
DO look for gross discrepancies that would disqualify a respondent. Do not expect a respondent's answers to always match the answers on his or her original screener.
DO, if possible, bring pictures of products to help respondents identify brands they use.
DO maintain respondents' confidentiality. Ask only for first names.
DO keep the length to one page.
DO thank respondents for being patient and responding to the same questions one more time.
DO notify the facility in advance about re-screening so that they can over-recruit if they are concerned about fallout as a result of re-screening.
DO consider adding one attitudinal "tie-breaker" question if more respondents qualify than you care to seat.