Ask and you shall receive

Editor’s note: Brett Plummer is research director at the HSM Group Ltd., a Scottsdale, Ariz., research firm.

While some companies are forgoing survey research during difficult economic times, others are enjoying the competitive advantage of understanding their changing marketplace through efficient survey research. Those organizations must strive to maximize their research investment by obtaining the highest quantity and quality of information available from their surveys. To accomplish this, it’s important to have clear objectives guiding survey construction and give adequate thought to each survey question. Otherwise, organizations run the risk of having the time respondents take to complete the survey plus the effort to collect, analyze and interpret the data outweigh the information gained from the survey.

Many people take survey question-writing for granted. But writing quality questions is an acquired skill that gets better with experience and guidance. Many problems with survey questions can be avoided with adequate survey pretesting, but unfortunately time and budget pressures can limit pretesting. In this article we’ve summarized some of the most important dos and don’ts of writing survey questions.

Do: Keep your objectives in mind

It is easy to get caught up in the wording and structure of survey questions, so much so that sometimes the ultimate objective of the question is forgotten. Always consider how you will use the information obtained from the survey responses.

Let’s suppose your objective is to find out the percentage of your customers that are receiving reports on time. While many survey researchers advocate using a 1-10 or 0-10 scale (e.g., “Please rate the timeliness of reports on a 1-10 scale”), how are you going to interpret the responses of 6, 7 and 8? A better way to reach your objective might be to simplify the question to, “Were you satisfied with the timeliness of the reports you received?” (yes or no). The data will provide you with a clear answer to your objective.

After drafting questions, ask yourself if the data for this question will give you the information you’re seeking. When multiple stakeholders contribute to survey development, survey questions sometimes end up being too long or too complex. In this case, it is helpful to go through the exercise of figuring out which questions are must-have and which ones are nice-to-have.

Don’t: Create confusing or ambiguous questions

Although your survey questions might seem perfectly clear to you, will your respondents understand them as well? Play the role of survey respondent when you think about how questions will be read and understood, or better yet, have somebody not close to the research topic take your survey, explain their answers and discuss with you any issues encountered. Will your respondents understand your questions as written and be able to answer them in a way to ensure meaningful results? You had better be sure or your data may be worthless.

Another pitfall that survey developers need to avoid is asking about more than one dimension within the same question. An example: “Please rate the timeliness and quality of the response you received when you called customer service.” Do you really want respondents to rate the combination of timeliness and quality at the same time? It would be much easier to interpret the data if you separated this question into two parts.

Be direct and to-the-point in your questions, avoiding ambiguity as much as possible. Only ask about one distinct topic at a time and avoid long question text that may erode your respondents’ interest. Avoid jargon, spell out abbreviations and acronyms and define potentially confusing terms. And this should go without saying, but we’ll say it here anyway: Proof and spell-check your survey!

Do: Consider which question type is best for each question

There are several different response types that will fit each question but there is probably only one that you should use. Figure 1 shows the main question types with examples. In addition, there are pros and cons to consider when using each question type, which we’ve summarized in Figure 2.

Don’t: Forget to carefully review response options for appropriateness and overlap

Response choices should be kept in a logical order. This will help respondents better understand the question and will help to prevent them from accidentally submitting an incorrect response.

When there is no logical order for response options, the order should be randomized. This removes any bias that response options may have by continually appearing first or last in the list of options. Keep in mind that the survey method (e.g., telephone or mail) can also contribute to the order bias. When presented with a list of response options, telephone respondents are more likely to select what they heard last (known as the recency effect) while respondents in mail/online surveys are more likely to select what they saw first (known as the primacy effect).

Your list of response options in all “choose one” and some “choose many” question types should be mutually exclusive and collectively exhaustive or the results obtained will be difficult, if not impossible, to interpret. If a single answer fits into two or more response options presented, then those response options are not mutually exclusive. Here is an example of a question where the response options are not mutually exclusive:

Which statement best describes your customer service call?

o My issue was resolved

o My issue was resolved with only minor hassle

o My issue was resolved after repeated phone calls

o My issue was not resolved

Do: Take into account how the data will be analyzed

Even though you think you’ve written a good survey question, ask yourself what kind of data analysis will be used and how you will use the analysis results. You may be surprised to find that there is a better way of getting the information you need.

Here is one example: Company X wants to assess purchasing intent in its customer base. Its marketing team writes this question: Using a scale of 1 to 10 where 1 is “not likely at all” and 10 is “very likely,” please rate your likelihood of purchasing Product A in the next year.

Now let’s suppose that the analyses reveal a mean score of 6.4, with 35 percent of respondents indicating high interest (8, 9 or 10) and 23 percent of respondents indicating low interest (less than a 5).

What should Company X conclude? Consider if it had asked this question: “Do you plan to purchase Product A in the next year? (yes or no).” The results of the simpler yes/no question would certainly give Company X a clearer idea of the overall purchase intent.

It is also important to consider how to structure questions that might be used for multivariate analyses. If you would like a concept to be part of a regression analysis, it is best to have the response options be separated by the same numeric interval (e.g., a rating question), in order to maximize the correlations between items.

Don’t: Lead your respondents toward answers

Respondents can inadvertently be led to answer questions in a way that confirms preconceived ideas of the researchers. It is the same concept as leading a witness in a courtroom. You want to give your respondents all available options for each question. Here is one example of leading respondents:

Given this economy, how likely are you to continue using Service X?

o Definitely will

o Probably will

o Probably will not

o Definitely will not

While the economy may be a valid consideration, not all respondents may see or be affected by the relationship between the economy and using a particular product or service. Instead, the question may be better worded simply as, “How likely are you to continue using Service X?”

Responses differ greatly when terms that suggest evaluative judgments or ideology are part of the question (e.g., “needy people” vs. “those on welfare”; “not allow” vs. “forbid” or “ban”). In writing your questions and responses, use neutral, non-judgmental wording in questions. If you have opinions on the topics you are asking about, the respondent should not be able to infer those opinions in the questions. And if possible, do not ask questions that respondents are afraid to answer (such as questions that might require an employee to review their manager), especially in the beginning of a survey. It may cause them to abandon your survey altogether.

Do: Include all valid response options

Where is the problem in this question?

What is your favorite color?

o Red

o Green

o Blue

Something critical is missing here, namely response options for respondents whose favorite color is yellow, orange, purple, etc. Of course all of these could be part of an “other” category, which is also missing from the response options above.

“Other” isn’t the only category that is often mistakenly left out. “None of the above” is another common omission. And for some questions, you should also consider “Don’t know” and “Not applicable” as valid responses. These omissions become especially problematic if responses are forced (often in an Internet or phone survey), where respondents cannot proceed to the next question without selecting a response.

There should be a response option for every respondent. If not, respondents may become frustrated and provide inaccurate data.

Also, make sure that your “Choose one” questions should not be “Choose many” instead. Here is an example of question that should be in “choose many” format.

What time of the day do you like to check your e-mail?

o Morning

o Midday

o Evening

o Night

o Check e-mail once per week or less

o Do not use e-mail

Don’t: Ask redundant questions

Our company has conducted hundreds of surveys over the years, and nothing seems to annoy respondents more than question redundancy. When survey respondents are annoyed, many will lose interest and speed through the survey. Even worse, they may intentionally submit incorrect answers.

The problem with many surveys is that even though the survey developer may not think there are redundant questions, the respondents may feel differently. Distinguishing between concepts is necessary for individual survey questions, but extremely subtle distinctions between questions will pass for some respondents as unnecessary redundancy.

Here are two questions with very small differences:

Please rate the value that customer reports add to your client-customer relationship.

Please rate the importance of creating reports for customers.

The wording may be different, but these two questions are essentially the same. And some respondents will recognize this.

Do: Consider where your question falls in the flow of the survey

When writing a survey question, give serious thought as to where in your survey it should be placed. Random placement of your questions might have a negative effect on the quality of your data.

Screener questions should always be presented first. If a respondent doesn’t qualify to complete your survey, don’t make them do any more than they have to. You might need them again in the future, especially if they are a customer, so don’t upset them!

Put the most difficult or most important questions first, before response fatigue sets in. Research has shown that data quality and completeness diminishes with longer surveys, and some respondents will feel that anything more than five minutes is too long.

Another consideration in question placement is possible bias from the ordering of questions. A respondent’s answer to a question might be different depending upon where in the survey the question is asked. If you are looking for a top-of-mind answer to an open-ended question, it may be best to ask the question early in the survey before important topics have been introduced that could influence the response.

It is generally considered good practice to group similar items together. This is a convenience to respondents to keep them from continually shifting their thoughts to different concepts. Grouping similar items may also elicit more information from respondents in open-ended questions and may help them differentiate between items when answering rating questions.

Maximum amount of information

In conducting survey research there are several important processes that eventually lead to data that will (hopefully) answer the questions you intend to answer. Without quality survey questions the resulting data might be very difficult to interpret, or even worse, completely useless. Thinking about the content, format, placement and analysis of each individual question will allow you to get the maximum amount of information out of your survey. With research ROI difficult to measure and defend, you’ll need to do everything possible to make the most out of your survey.