Questionnaire design from the trenches
Editor’s note: Srijana Dhakhwa is managing partner of Stanford ResearchTech, a Concord, Calif., consulting firm.
Interviewing an executive is a performance that takes intellect, perception and boldness. There are no guarantees on how the conversation will unfold but a well-scripted questionnaire goes a long way in producing an interview that you will be happy with. This article is about how to best script your questionnaire so you get the most out of your conversation with any executive.
Executives are individuals with strong egos and equally strong opinions. Rather than merely surveying executives, you conduct guided dialogues with them. Having a dialogue makes it an equal exchange; the executive feels gratified for getting something from the conversation and is much more likely to open up about key areas of information.
Thoughtful preparation of the questionnaire with the executive respondent in mind will also be more effective in addressing the key areas that are of concern to your clients. There are ways to structure the questionnaire so you get the most from your conversation with the executives.
• Rapport
Building good rapport is extremely important in depth interviews. It makes the respondent open up and makes him or her feel comfortable. Someone who feels comfortable is likely to share more information. There are ways to build good rapport with a respondent right away. The following approaches have proved to be very effective.
a) Ask about his or her role within the organization. This gives the respondent the chance to talk about him or herself. The discussion about the role within the company exposes the respondent’s responsibilities so that you can gear the questions towards those areas. Being more focused on the respondent’s responsibilities will also keep him or her more engaged and will help you get more information.
This is also a very safe way of gauging the respondent’s style of conversation and mood, further aiding rapport-building.
b) Ask about successes. This works very well in establishing rapport. It gets the executive to focus on his or her key areas of achievements in the last year and elaborate on how his or her group did. People like talking about their own successes and this question makes the executive share a lot of information about the priorities and how well those were achieved.
c) Ask about challenges. After the executive has talked about successes, ask the individual about the challenges his or her team currently faces. The information that the executive otherwise might not have shared is more easily obtained if the question on challenges is asked after the question on successes. People seem to be more open to discussing the negatives just after they have basked in their own positives.
The question on challenges also smoothly opens up dialogue about the key focus areas for the coming year.
d) Drill down into details. Now you can drill down into details about areas that you are interested in, whether they be operational strategies, marketing strategies or other issues.
In each section of detailed questions, however, stay with the basic format of asking positives before the negatives.
• Logic
The importance of logical flow in a questionnaire cannot be overstated. Any misstep in logic, which might be forgiven when on paper, can haunt you during the interview. So each question has to have a reason and each question needs to be logically related to the one prior.
When conducting an interview you are tapping into the respondent’s natural thought process. This thought process is initiated by your opening remarks or the first set of questions you ask. A well-constructed and logical questionnaire stays with this natural thought process, thus extracting the optimum amount of information from the individual. Jumping away from the respondent’s thought process takes you into dry territory and you end up with little useful information.
Do not confuse a questionnaire for a depth interview with a quantitative survey questionnaire. I see this often. A researcher wants to make the depth interview questionnaire like a survey tool. Anyone going through market research 101 knows that results from depth interviews (unless one is lucky enough to have a lot of them) should not be used to make quantitative conclusions. Depth interviews are to gather information on ideas, trends, issues, etc.
Having said that, it is very common to use multiple-choice questions, particularly questions to rate and rank information in depth questionnaires. It is crucial that these multiple-choice questions are properly placed. The best places are at the beginning or the end of a section, depending on how important these questions are. The thought pattern needed for multiple-choice questions is different from the thought pattern needed for open-ended questions. Thus, placing a multiple-choice question in the middle of a concept can disrupt the thought process of the interview and thus prevent good information flow. It is also essential that the question itself as well as the choices in the multiple-choice question flow with the logic of the questionnaire.
• Screening
Keep questions related to screening or qualifying to a minimum when talking to senior executives. Executives want to conduct a discussion, not be surveyed. Nothing says “survey” like a series of qualifiers.
• Focus
Every concept has to be distinct and clearly differentiated otherwise it will confuse the respondent. For instance, a question such as the following leaves the respondent confused and unsure as to what to focus on:
What are the potential challenges of using mobile organizers or laptop computers?
It is not clear whether the respondent is supposed to address mobile organizers or laptop computers or both. You have to separate these two subcategories.
If providing options, make sure they support the main question. You want each question to be focused on the main theme. Options that do not support the main question will distract the respondent.
Here is an example of what not to do:
Are you involved in the investigation decision to evaluate the use of telecommunications solutions?
Yes, I am involved in the investigation and/or selection process to outsource additional agents
Yes, I am involved in the investigation and/or purchase decision for new technology solutions
No
An alternative way of asking the same question is the following. Here the options support the main question, rather than reiterating the question in different forms.
What is your involvement in the purchase process for telecommunications solutions? Check all that apply.
__ Involved in the investigation to outsource additional agents&
__ Involved in the selection process to outsource additional agents
__ Involved in the investigation for new technology solutions
__ Involved in the purchase decision for new technology
__ Not involved
• Eliminate redundancies
Redundancies can kill rapport with the respondent and at times bring the interview to a halt. Eliminating redundancies is both a science and an art. It is a science because we can simply eliminate redundancies by consolidating similar questions. It is an art because respondents might address issues in the course of the discussion that are later specifically included as a question. It takes an attentive and skillful interviewer to avoid redundancies in such cases.
For example, asking the following three questions about the same solution is redundant.
What, if anything, is most compelling about this solution?
What benefits would you hope to achieve with such a solution?
What would be the key criteria you would use to evaluate this type of solution?
A product is compelling to a buyer because of the benefits and a buyer uses the most important benefits as the criteria for evaluating a purchase. These three questions will elicit the same answers.
Here is an example of a potential area of redundancy that requires experience to avoid:
What is your reaction to that solution?
What is most compelling about this solution?
What is least compelling about this solution?
If the response to the first question addresses either of the other two, the interviewer has to know not to ask for the same information.
• Phrasing
Some ways of phrasing questions call for open-ended, opinion-based responses. Some call for crisp, multiple-choice-type responses. The wrong phrasing can lead to a very ineffective discussion.
For instance, when you start a question with “what, why, how, who, when” the respondent’s mind instinctively wants to give an opinion and would start forming one. You will stifle the flow of conversation if you follow a question that starts with one of the above with a list of multiple choices.
Here is an example:
What type of a structure have you implemented?
Structure A
Structure B
No preference/ideas
The following rephrasing of the question is more effective:
Which of the following structures best represent what you have implemented?
Structure A
Structure B
No preference/ideas
Similarly, the following question prepares the respondent for an opinion:
Please describe the breadth of your service coverage:
All regions
Northeast only
Southeast only
Other
Rephrase it as follows:
Which of the following best explains your service coverage?
All regions
Northeast only
Southeast only
Other
Guided dialogues
Treating depth interviews with executives as surveys will not get the most effective results. Treat these interviews as guided dialogues. A tightly scripted questionnaire will help this dialogue proceed smoothly and will increase the chance of an effective interview.