Editor’s note: The following is an excerpt from a chapter in the U.S. Dept. of Commerce report "Approaches to Developing Questionnaires.” Principal contributor: Anitra Rustemeyer Streett.

One way to evaluate questionnaires is to investigate respondents' understanding of the intent of specific questions and the meaning of their replies to those questions. This technique is called frame-of-reference probing, and is done by asking the respondent some additional questions. It is designed to address concerns about whether the questions, definitions, and instructions proposed for a questionnaire convey the frame of reference desired. Probing to determine respondent frame of reference can be especially useful when words (like "crime") that are key elements in a survey are thought to carry emotional impact.

The probing questions can take different forms: either ad hoc questioning by the interviewer or administration of a set of questions written in advance (called "structured"). Ad hoc questioning usually takes place after the survey questionnaire has been administered. When structured follow-up questions are asked, the probing might be done immediately after the question containing the words or concept of interest is asked; alternatively, it might be done after the survey questionnaire has been completed.

Frame-of-reference probing can be incorporated at various stages of the questionnaire development process. It might be planned as a part of a pilot study or field test or it might be done during the actual survey.

Personnel and skill requirements

This technique is implemented by interviewers, and to some extent, the skill requirements involved depend on whether the probing takes the form of structured follow-up questions or unstructured questioning. In the former case, regular interviewing skills are required; in the latter, more extensive interviewing skills such as detailed probing, the ability to think quickly, and others are also necessary.

When this technique is used during informal testing, it may be preferable for researchers/questionnaire designers to conduct the interviews to give more insight into respondents' interpretations of the word or phrase of interest.

Selection of respondents

The way respondents are selected for frame-of-reference probing depends on which stage in the questionnaire design process the method is used. During the questionnaire development process, respondents are selected using the same purposive selection strategies as those used in informal tests or unstructured interviews. If respondents' interpretations of questions in formal tests or actual surveys are subjected to investigation using this technique, however, respondents have already been selected through scientific procedures. Depending on time and resource constraints, everyone in the sample can be included in the frame-of-reference probing, or respondents can be subsampled and the additional probing questions asked of only a percentage.

Preparation

In advance of data collection, the following basic decisions need to be made:

1. Decide when during the questionnaire design process to probe respondents’ interpretations.

During the questionnaire development, probing to determine respondent frame of reference for key concepts can facilitate improvements in question wording and thereby avoid collecting data that cannot be properly analyzed statistically. This type of question investigation can warn the survey designer of ambiguities that will cause respondent confusion and irritation. If ambiguities concerning the meaning of questions are present, it is likely that the interviewers will be asked to explain what is meant or what type of answer is wanted. When interviewers are asked to explain questions, the chance of interviewer bias increases dramatically.

If probing to determine respondent frame of reference is included in the final questionnaire used for the survey, it can help to illuminate the answers provided in the survey. The answers to the probing questions may help the survey analyst to understand what appear to be inconsistent answers. And in a repetitive survey, problem questions can be deleted or changed for subsequent interviews.

2. Decide which words or phrases to probe.

Words or phrases that are central to collecting uniform information and thought to be susceptible to misinterpretation should be subjected to investigation. For example, in a study designed to evaluate the seriousness of various crimes, the respondent might be asked to rate the seriousness of an event described as "An offender injures a victim and the victim dies." To know whether the respondent answered in general terms or attributes specific circumstances to the event before rating its seriousness, additional probing should be done to determine how each respondent interpreted the question.

3. Decide where in the interview to probe.

If the questions added for the frame-of-reference probing do not disrupt the interview (by changing the subject, for example) and are not expected to bias the remaining survey questions, then it is probably best to ask them immediately after the question where the word or phrase of interest appears. By placing the probing questions immediately after the survey questions of interest, there should be no doubt as to what word or phrase is being referenced. If the probing questions might disrupt or bias the interview (such as detailed questions about sources

of income, traffic accidents, or the nature of mental illnesses in the family), those questions could be placed near the end of the interview and preceded with a transition statement such as "Earlier I asked you about…; now I have just a few more questions about that."

4. Arrange probing so that only a few questions (two to four) are probed with a respondent.

The number of survey questions to be investigated by frame-of-reference probing is decided by the researcher. However, if more than about two to four words or phrases are to be investigated, it might be better to limit the number subjected to probing with anyone respondent and interview a larger number of persons to collect enough data. Important considerations in setting the number of questions to be investigated are the total length of the interview and the respondent's tolerance for being questioned in detail on subjects for which he or she may have little interest and/or knowledge. Unless the respondents selected for this type of interviewing are known to be especially knowledgeable or interested in the topics to be probed, it may be best to assume a low level of knowledge and interest and arrange the probing questions accordingly.

5. Determine how many and what kind of probes to use to investigate each word or phrase under study.

The optimal number of questions used to determine the meaning attached to a word or phrase is probably about three to five. If too few probes are used, there is the risk of superficial or inadequate treatment of the subject; if too many are used, there is the risk of being tedious, appearing to challenge or question a respondent's views, beliefs, or attitudes, or of appearing to be administering a test in which these are "right" and "wrong" answers.

Clearly, adding questions to an interview results in a more time-consuming interview. In addition, there may be some respondents who will dislike being asked to report information such as what they were thinking about when they answered a question, or other questions requesting them to think about how they think about things. If the probing questions are carefully worded, it should be possible to avoid putting respondents "on the spot." An illustration of a question that was carefully constructed to avoid putting a person "on the spot" is: "Speaking of crime, everyone agrees some acts are crimes, but there are different ideas about others. Do you believe it is a crime for someone to...?"

6. Arrange a method of probing and presentation of additional questions.

The method of probing depends on the stage of the questionnaire design process at which the technique is used. When it is used for questionnaire development, it might be more useful to the researcher if interviewers are given guidance on what information is desired and then allowed to develop their own follow-up questions. To some extent, the choice between structured and unstructured methods during developmental work depends on the level of experience of the interviewers; less experienced interviewers and those not familiar with research methods may require more structured assignments. If used during the survey itself and if all respondents are to be asked all frame-of-reference probing questions, the follow-up questions should be printed on the questionnaire so that they will be asked in the same way, and at the same time during the interview, of all respondents.

7. Establish a system to record results of the probe.

Two common ways of recording results of unstructured interviewing are tape recording and having a second person accompany the interviewer to take notes. If structured questions are used (with printed questions to be read), then precoded answers may be developed to aid the interviewer in recording the answers quickly.

8. Develop a technique for reconciling survey question response with probing response if the two answers are expected to be the same.

Sometimes the frame-of-reference probing questions ask for the same type of information as the survey question, but in a different manner. When the same type of information is asked, the respondent may seem to give quite different or contradictory responses to the frame-of-reference probing than he or she did to the survey question. Reconciliation of responses is important for these cases. If this happens, the interviewer might say, "In light of what you've just been saying, I'd like to go back and ask again one of my earlier questions; ...(repeat question)."

Operation

Since frame-of-reference probing is generally done in conjunction with one of the stages of testing or with the survey itself, the selection of a site and other operational details are taken care of in planning for the main event. Some additional details may be necessary to accommodate the use of this technique, however. For example, if experienced interviewers rather than researchers are involved, they may require extra training on how to ask the additional questions. If unstructured probing is required, the training may be longer, more complicated, and different in content than if structures questions are added to the questionnaire.

If a decision is made to use frame-of-reference probing questions for a subset of respondents rather than for all of them, additional interviewer instructions may be necessary.

Data analysis is the final step in the operation of frame-of-reference probing. Analysis focuses on responses to the probing questions and may also include their relationship to some of the other subjects of interest in the survey. Take, for instance, the example cited earlier in which respondents are asked to consider the seriousness of the following statement: "An offender injures a victim and the victim dies." Do people who imagine the injury to be inflicted during a barroom brawl rate the seriousness of the crime the same as or different from people who imagine it to have been the result of a traffic accident? Differences in the responses of male versus female respondents or consistencies in the pattern of a single respondent's replies to a variety of such vignettes may also be of interest. If there is no differentiation among the ranking of crimes which are considered quite different by the questionnaire designer, there may be either a problem with the language in question (suggesting that the wording should be changed), a problem with the researcher's notions about the seriousness of the crimes (suggesting that different examples be included), or perhaps a problem with the respondent's ability to make the desired distinctions (suggesting that the questions should be deleted). Such an analysis conducted in the conjunction with the final survey may provide explanations for some of the results from the analysis of the survey data.

Time considerations

For the most part, the time required for planning and executing frame-of-reference probing overlaps preparation for the survey or test to which it is being appended. The selection of the testing vehicle, the data collection, and the data analysis all occur simultaneously with operations for the test or survey. Thus, the additional time necessary to use this technique is minimal. Drafting the probing questions (or deciding what information is required from unstructured probing) cannot take place until after the questions containing the words or phrases of interest are written, and it must be done before the interviewers who will administer the questions are trained.

Analysis of the information collected from unstructured frame-of-reference probing may take longer than from structured probing, since an additional coding phase may be required.

Cost considerations

In general, the cost factors involved in frame-of-reference probing, over and above those of the test or survey itself, are slight. Additional expenses may be incurred for reproduction of questionnaires or interviewing materials, interviewer salaries for longer interviews, and salaries for the researchers/questionnaire designers. If members of the research staff conduct the interviews, cost of travel and related expenses, and extra salary expenses will also be incurred.

Mode of data collection

Frame-of-reference probing is suited for use in designing interviewer-administered surveys, either face-to-face or telephone. It could also be used in a face-to-face test of a mail questionnaire, but mail questionnaires themselves are not well-suited to the technique. Structured follow-up questions could be incorporated into a mail questionnaire, but since the respondent is free to answer questions in any order and over a long period of time, the responses to the probing questions may not be good indicators of what respondents had in mind when answering certain questions.