SPONSORED
Optimize survey design to collect accurate data
By Chris Benham, CMO, Alchemer
Researchers often jump directly into the “build” phase of the survey but once you’re writing questions you’re nearly halfway done with your project. Survey questions are like the walls, floors, doors and windows of a house: They’re vital, but if you put them up without laying a foundation you’re in for some serious trouble. Time to create a great data collection survey.
Constructing survey questions
Survey design involves thinking about the psychology, emotions and words behind the questions. The design process is the strategic phase. It should include your survey goals and learning objectives.
Building the survey, on the other hand, is the tactical phase. It considers logistical issues like security, logic, survey fatigue, bias and data collection. When building surveys you think about the details, including question types, survey length, anonymity and data analysis.
Turning goals into real data collection survey questions
Most survey-building teams consist of one to three people; however, they should always consider stakeholder input during the process. Data collection survey design – particularly the identification of goals and objectives – can be a group effort but having too many workers on a construction site can create more problems than it solves.
Ideally your survey construction team will be made up of the people who will be responsible for presenting the data that has been collected, because they need to understand the source to present outcomes more clearly. Other good team members may be those who are going to act on the data. They’ll often have different insights into question phrasing and order.
Qualitative vs. quantitative data collection
One of the biggest questions about questions is whether to use qualitative or quantitative question types. The answer depends on what you want to achieve. Qualitative questions are open-ended. They usually include a “why” somewhere and they can be very useful in helping to define a problem. Quantitative questions are designed to simply gather data, not ask opinions.
Both are highly useful but you should choose between them carefully because they offer very different kinds of data collection and outcomes.
Qualitative questions help define a problem. They ask “why?”
If you are exploring a hypothesis, a qualitative survey can identify a problem and its nuances before conducting a quantitative survey. Qualitative questions are open-ended, which means text analysis is required to interpret results (and these are particularly susceptible to interpretation bias).
Qualitative questions are always open-text questions but they come in many forms, including:
- Text box: Responses can be one word to one sentence long.
- Essay box: Several sentences to several hundred words, depending on the limit you set (or don’t set).
When asking a qualitative question, consider using an autocomplete feature to minimize data cleanup. This lets you suggest common answers so that you get consistent responses but it can also introduce bias, so proceed with caution. It’s worth trying, however, because having the same answer format will make it much easier to analyze the data. For example, these answer options all mean the same thing to a human respondent:
- twenty-five
- 25
- Twenty-Five
But each format (including uppercase versus lowercase) will be treated as a different answer option during open text analysis. To get the full impact, long-form questions really need to be read individually.
Quantitative questions, on the other hand, ask “what,” “when” or “how.” These questions typically quantify a predefined problem so you can understand how prevalent it is. Quantitative questions have limited answer options, which makes it easier to measure the results.
These are the most-used quantitative survey questions, and your respondents will quickly recognize the type. There are less common, more advanced varieties of quantitative questions as well, but keeping data relatively simple will create a better experience for those answering your survey. Common data collection question types include:
- Radio button: Use these when you want a single-answer option.
- Checkbox: Use these when multiple answer options are acceptable.
- Drop-down: These are most used as single-select but can also be used as multi-select answer options.
- Likert scale: These are most used for measuring emotions such as satisfaction or agreement. Odd-numbered scales allow for a neutral response.
Avoiding survey fatigue
Online data collection surveys shouldn’t be exhausting. Tired respondents will either abandon your survey or give you substandard data. When you’re building your survey, you need to take the respondent’s experience into consideration.
Having a long list of items to rank generally increases fatigue and dramatically increases drop-offs. Many surveys fail to collect useful data simply because they were not designed to keep their respondents interested. As a survey builder, it is your job to reduce survey fatigue whenever possible while still gathering solid data that your team can act on.
General question guidelines
You also need to make sure that people taking your survey don’t get thrown by awkward answer options or question construction. A common error is creating overlapping answer options. Selecting one choice should completely exclude all the others. When answers intersect it can cause a lot of confusion. For example:
Correct format:
How long have you been a member?
1-10 years
11-20 years
21-30 years
31+ years
Incorrect format:
How long have you been a member?
1-10 years
10-20 years
20-30 years
30+ years
You’ll also want to refrain from using double-barreled questions, which combine multiple questions into one. This adds confusion and skews your data. For example:
Don’t ask:
How satisfied are you with our buffet food and drink options?
Instead, ask:
How satisfied are you with our buffet food options?
Address drink options in a separate question. If you’re not sure that you’ve included all the possible responses in your answer choices, include an “other” option for those who don’t find the right choice in your list. A forced answer that doesn’t apply will taint your results.
The importance of survey validation
Validation is the process of checking your survey to ensure it meets your specifications and fulfills its intended purpose. Like editing a document, validation requires a detailed review of answer options, logic, reporting values and reporting data to verify that you are set up to collect quality data.
How to validate your survey
Validation is a key component of great survey design but it’s often overlooked as people skip straight to testing. Testing can uncover some problems with a survey but validation is a more rigorous review process.
A common problem occurs when different data formats are treated as different answer options. This will make it hard to analyze the data unless we do some data cleaning to standardize the answer format. Changing the question type, adding instructional text on the proper format you would like entered, using autocomplete or using a data validation feature will go a long way in solving this common problem.
Testing your survey
Testing your survey simply involves taking your survey on the different devices that your respondents will be using to ensure it displays and flows correctly. Advanced survey tools provide a testing feature that quickly generates test data so that you can also look at the results to see if it reports as you expected. Run a few reports on the data, then ask yourself these key data collection questions:
- Are your questions reporting the way you expect?
- Are you able to create the reports you need using the data you’re collecting?
- Is the data in the format you need?
The power of survey logic
One of the coolest parts of building a survey is adding logic. Put simply, logic is a set of conditions that you can apply to a question, answer or even an entire page of your survey that affects how it performs. Survey logic is extremely powerful and its benefits come in two flavors:
Fatigue fighting: Keep respondents engaged by only showing them questions that are relevant to them.
- Page-jumping: Skip entire pages that aren’t relevant to a respondent.
- Show-when logic: Only show questions when conditions are met.
- Percent branching: Randomly assign a set percentage of your respondents to a branch of your survey.
- Piping: Inserting data collected early in the survey into a later question.
Bias fighting: Avoid any bias that might come from your question or content order.
- Randomization: Randomize question and/or answer options.
- Disqualification: Prevent those who don’t qualify from answering your survey to collect cleaner data.
- Survey timing: Identify and disqualify survey responses that were answered too quickly.
- Vote protection: Prevent respondents from taking your survey more than once.
Survey logic is one of the best ways to keep your survey relevant and collect quality data.
A well-built survey drives successful data collection
Approaching your survey build with care and attention will make sure that it serves your ultimate purpose. By creating an appealing design and an experience that’s as personalized as possible, you’ll get more engaged respondents who give you better data. Keep these best practices in mind and you’ll be well on your way to effective data collection.