A U.S. Department of Commerce report, “Approaches to Developing Questionnaires,” includes a section on a technique which might prove especially helpful to those involved in Business-To-Business research. A summarized excerpt from this section appears below.
Sometimes a questionnaire designer is required to develop a questionnaire on a topic about which little is known, and about which little information related to questionnaire design exists from previous surveys. In this situation, the development of a questionnaire can benefit from the use of unstructured interviews with members of the intended respondent universe. The term “unstructured interview” is used here to describe a discussion of the proposed survey topics between a member of the target survey population and the questionnaire designer. The discussion is guided by a topic outline rather than a set of specific questions. When sufficient numbers of such interviews are conducted with respondents who are fairly representative of the target population, the technique can provide ideas and insights about how best to structure the questionnaire before the first draft is written.
It is a particularly valuable technique when there are many divergent interests in a survey. When there is more than one sponsor, initial disagreements can exist about what kinds of information can and should be obtained. This technique transfers the questionnaire design decisions from dependence on the tastes or preferences of the survey sponsors to reliance on the results of the field process employed.
Several questionnaire design issues can be addressed by using this technique. The specific uses of unstructured interviewing include:
- Topics previously thought to be important for inclusion can be discarded as unnecessary or irrelevant, and topics which had previously been neglected can be identified as important in fulfilling the objectives of the survey.
- A determination can be made as to whether the information requested in the survey is readily available to respondents and whether particular kinds of questions can be asked.
- An evaluation can be made of which topics might be especially sensitive to respondents.
- Assistance can be provided to determine how to phrase particular questions so that the vocabulary is familiar to respondents and the words mean the same things to all respondents.
- Decisions can be made concerning the preferability of open- vs. closed-ended questions to obtain particular types of information, and a range of answer categories for closed-ended questions can be specified.
- An identification can be made concerning who in business is in a position to respond most accurately to questions on the survey topics and therefore, would make the best respondent.
- Suggestions can be made concerning the optimal order of questions or survey topics.
- Insights about which aspect of a topic appeals most to people may be used to determine the best way to approach respondents in order to encourage their participation.
Method
Personnel and skill requirements
A key concept in the successful use of unstructured interviewing is flexibility. The questionnaire designer functions as a researcher during this process, and must keep the objectives of the study firmly in mind while dismissing any fixed ideas about how to structure the questionnaire.
Best results are achieved when several people, including one who serves as team leader/questionnaire designer, work together as team. The team should include interviewers as well as data processing and subject matter specialists if at all possible. This allows diverse ideas and insights to be used in the refinement of the survey instrument.
Persons selected to conduct unstructured interviews should be experienced interviewers and be capable of understanding the broad perspective of the research project for which the questionnaire will be designed. This type of interviewing requires skills different from those for structured interviewing (i.e., interviewing in which questions are read verbatim from a questionnaire), and only some interviewers on a regular field staff are likely to possess these skills.
Interviewers selected for this type of assignment should feel comfortable “thinking on their feet” as they will not have a questionnaire script to use as a crutch; if they are easily flustered or confused, they give respondents the impression that they are incompetent or that the study is unimportant. Members of the interviewing team need sufficient experience in unstructured interviewing to be sensitive to the effects of wording changes and to recognize responses that indicate potential problems with question wording or order. In addition, interviewers should be ale to tolerate long pauses while the respondent thinks or looks for answers, have the ability to probe nondirectively to get the respondent’s ideas, and have a thorough understanding of potential problems in questionnaire design which can affect the achievement of survey objectives.
Some of the interviewers should be highly knowledgeable and skilled in structured interviewing techniques. This allows informed judgments to be made concerning the kinds of things which can be asked in a close-ended format and what topics respondents can be expected to respond to within a structured interview.
Unstructured interviewing is actually a combined data collection and analysis process. In addition to the interviewing skill necessary for successful results, a “coder” who is capable of making independent judgments is an essential part of the process. This person should be able to analyze and tabulate results of the previous day’s work while the interviewers are in the field conducting additional interviews and then meet with them to explain how and where they are failing to meet survey objectives. The simultaneous conduct of these two tasks speeds up the questionnaire refinement process.
Finally, sponsors or subject matter specialists can provide valuable insights in the frequent meetings held to charter the course for the work.
Selection of respondents
Respondent selection for unstructured interviews generally involves purposive rather than systematic sampling. Although rigorous scientific selection procedures are not necessary, respondents should be members of the population to be surveyed and should be fairly representative of that population.
The characteristics of people asked to be respondents for unstructured interviews may depend on the survey topic. For example, in developing a questionnaire dealing with saving habits to be administered to a national cross-sectional sample, the initial round of developmental work may include interviews with people from a variety of demographic population subgroups. During additional interviews, however, different classifications of saving habits may emerge, and it may be necessary to locate and interview persons who are members of specific categories. Thus, the “sampling” of respondents is an iterative process, too—as is the questioning of these respondents.
Respondents may be located by contacting business or community organizations.
Preparation
Before embarking on this phase of a questionnaire design project, the team leader should become familiar with the objectives of the study and make a list of the data elements which are considered necessary to meet those objectives. These data elements include topics and concepts which are particularly vital to the quality of the study, or are otherwise thought to be related to the survey objectives.
Prior to the first discussion with a respondent, the team leader should prepare some alternative orders in which the topics might be discussed, as well as any specific words or phrases to be used in relation to any particular topic.
The team leader’s next task is to develop the work sheets to be used by the interviewers and coders. Those serving as interviewers should review the materials and meet with the team leader to discuss study concepts and objectives. The interviewers need to be provided guidance, so they will not go beyond the scope of the project.
Operation
Interviewers may begin each interview by explaining that they are working on a very early phase of preparing a new survey. They should emphasize the reasons for, and importance of talking to people before a questionnaire is prepared.
During this type of interviewing, the interviewer should follow up on answers or comments that seem to have a bearing on how a concept is interpreted by the respondent or how a sequence of questions should be ordered. The interview should have a conversational flavor rather than a question-and-answer format of a formal interview. Interviewers should understand that their objective is not to collect data in the usual sense—rather, it is to become aware of the difficulties that are likely to arise when the survey is being conducted.
Throughout this process, extensive note-taking is valuable, so that insights gained during an interview are not lost or confused with other interviews. Verbatim recording, by shorthand, speedwriting or tape recorder is ideal for this purpose.
Even abbreviated note-taking can make it possible for an interviewer to return to statements made earlier by the respondent. Following up immediately on some statements could take the interviewer off the topic being pursued, but “passing remarks” and apparently contradictory statements by the respondent can provide additional insights on how to phrase survey questions.
During each unstructured interview, the interviewer should record how each key inquiry was phrased, as well as the wording used by the respondent in answering the question. (Since interviewers often respond to the answers of respondents with idiosyncratic or instinctive phrases of their own, it may be more difficult to remember their own words than those of the respondent.) Notes should be made (during or immediately after the interview) concerning the ordering of the inquiries (if different from the outline), how on topic relates to the next, if and how they overlap, what effect topic order has on the flow of the interview, the respondent’s reaction to specific questions of interest, and the apparent level of difficulty of the inquiry for the respondent.
Tape recording, with the respondent’s permission, can be useful as long as time is available to listen to the tapes and extract information from them. Ideally, the team leader, team members who are conducting the unstructured interviews, and coders should meet frequently to discuss what they have learned to date. The reason for these frequent meetings is to allow all interviewers to gain insights from the experiences of the others and to help one another interpret respondents’ comments. Under the guidance of the team leader, changes to the topic outline should be made to refine ideas on how to present topics and sections of the questionnaire, and the order in which to present them. As experience using the topic outline is gained, interviewers will develop their preferred question wording for topics. They should exchange those wordings during their meetings and then try the wording used by others in successive interviews.
The input of the coder is beneficial in noting ambiguities or superficiality in the responses obtained in previous interviews which require further clarification before the response can be coded. Also, the relative frequency of responses to open-ended questions, the range of conditions imposed by respondents on their answers (e.g., “It depends on...”), and potential response sets can be obtained from the coders’ tallies. The coders’ analyses and the interviewers’ annotated transcripts are discussed among team members, patterns are identified, and suggestions are made concerning potential question formatting, sequencing, etc.
No set number of completed unstructured interviews or days of unstructured interviewing can guarantee a good questionnaire. Perhaps the best indicator that enough unstructured interviewing has been done is the lack of new insights and ideas on question wording and order by team members. The responsible researcher (i.e., the team leader) must judge whether the team has fulfilled its mission, and when the process of putting together the first draft of the questionnaire should be undertaken.
Time considerations
The process outlined here may take longer to complete than drafting a questionnaire without any field work. On the other hand, when the questionnaire is drafted after these procedures have been followed, it is likely to require far less modification; therefore, time required for unstructured interviewing may be wholly or partly recovered later. The exact amount of time involved depends on the number of people who are available to conduct interviews, the number of interviews completed daily by each interviewer, and the iterations of the topic outline, question wordings, etc., required before members of the questionnaire design team are confident to construct a questionnaire.
In general, when the use of unstructured interviewing is incorporated into the development process, two to six weeks should be allowed in the time schedule. This includes the preparation time for the team leader as well as the interviewing time itself. It does not include completion of the initial questionnaire draft, which would be required regardless of whether or not this technique is used. However, drafting the questionnaire should be much less time-consuming,
because the knowledge gained from the unstructured interviews will clarify concepts and resolve most of the issues that are typically debated; e.g., which words to use and which to avoid, how much detail to request of respondents, and the order in which to present topics.
Cost considerations
The monetary costs associated with the use of unstructured interviews are essentially limited to the salaries of the personnel who are members of the team. Depending on the number of people involved, the number of interviews conducted, and the amount of time spent in analyzing the interviews, these costs could vary considerably. In addition, other expenditures may be necessary for travel if the interviewing site is not located near the cities of the people working on the project.
One other “cost” should be mentioned here: the burden on the audience surveyed. Although unstructured interviewing places some response burden on this group, this investment may be more than repaid later if the unstructured interviewing results in a more efficient questionnaire than would be prepared without this type of field work.
Mode of data collection
Regardless of whether the final survey will be conducted face-to-face, on the telephone, or by mail, the use of unstructured face-to-face interviewing can provide valuable insights on how people respond to the topics of the survey. Benefits accruing from establishing the relevance of specific topics to the survey objectives, defining key concepts, and identifying words which have similar meaning for all types of respondents will be equally pertinent for surveys conducted through any method.
Some of the other insights gained through the use of this technique, such as the specification of question order, may be unique to the mode in which the data are collected. If the final survey is intended to be conducted exclusively on the telephone, unstructured telephone interviewing could conceivably be conducted.