That could hurt

Editor's note: Alice Blackwell is vice president of MDC Research, Portland, Ore.

Most people don’t rely on their own personal knowledge of dentistry to conduct dental exams and treatments. Instead, they go to a dentist. Similarly, most businesses don’t count survey design as a core competency but they often attempt to do DIY market research. The research industry has seen some truly remarkable innovations over the last decade, including online survey technology and other SaaS offerings which allow anyone easy access to the mechanics of survey creation. What these tools don’t do, however, is teach the user how to write an appealing and scientifically sound questionnaire, nor do they point out design issues with surveys created on the platform. There is no “survey check” that puts a squiggly red line under problematic question wording or formatting.

It’s a specialized skill to be able to understand the analyses needed to accurately address the objectives, then envision the structure and type of data that will be required to perform those analyses.

The temptation of “cheap” survey access, plus lack of professional guidance, has led many organizations down a path fraught with peril and even caused some to make fatally flawed decisions. Most DIY survey platform users don’t have enough training in the theory and practice of survey design to create an instrument that will yield the data they need to make optimal, informed decisions. There’s nothing wrong with acknowledging that you don’t understand all aspects of questionnaire design, just like there’s nothing wrong with knowing that you don’t know how to safely and painlessly extract your own tooth. Both are complex skills that require extensive training and tackling the job yourself is likely to lead to unintended (and quite possibly disastrous) results.

The DIY survey is especially rampant due to the sheer volume of surveys in modern life. Everyone is exposed to multiple, ongoing requests to participate: there’s the survey request printed at the bottom of your retail receipt; the pop-up survey that appears on your favorite Web sites; the phone call from a pollster or market research agency; the sincere request from a charity to complete a mail survey; the list goes on and on. When the average person is exposed to so many surveys (both well-written and poorly written), some begin to believe they understand exactly how they work. For those who have taken part in dozens, or even hundreds, of surveys over the years, it’s tempting to believe they know enough to write one from scratch. But mere exposure to surveys is not comparable to formalized education and experience in questionnaire design.

It costs more, and takes more effort, to clean up or revise a poorly written survey than to write a good one from scratch.

I studied research methodology for six years in college and take part in industry organizations, forums, continuing education and conferences throughout the year. My firm has several such full-service research consultants who work together to optimize the work we do for our clients. We design and execute more than 500 surveys a year and we’ve been doing so for almost 40 years. Even with all that experience, we still provide critical feedback and improvements to one another’s surveys and passionately debate the best way to elicit the necessary data. Survey design is a major part of my specialized skill set and one in which I, and other researchers, take great pride.

For those who haven’t formally studied survey methodology, writing your own survey is a bad idea. Here are the top six reasons why:

1. You’re too close to the topic.

If you’re considering writing your own survey, it’s probably related to a topic that is important to you or to your ability to fulfill your professional obligations. But your investment in the topic and the importance of the outcome to you are potential stumbling blocks. It’s all too easy to allow your own biases to creep into question wording. When you pass a draft around your office, your internal team is not apt to recognize instances when your corporate culture and/or jargon are skewing the meaning of a question for respondents. You’re also unlikely to recognize all the opportunities for eliciting negative ratings and comments or how to use them in a constructive and unbiased analysis of the final data.

2. You’re overburdening your respondents.

It’s very tempting to go overboard on survey length. The thinking is, “While we’ve got their attention, let’s also ask X, Y and Z.” This tendency to keep expanding the survey in the name of efficiency leads to respondent fatigue, poor data quality and survey abandonment. You need a survey that is custom-designed not only to address your questions and objectives but to be appealing and relevant to respondents. The rare customer or potential customer may share your fascination with the minutiae of the purchase decision process for your product or service but the vast majority do not. They may like your product and they may even love your brand but that doesn’t translate into wanting to spend 20-30 minutes answering detailed and repetitive questions. Factors like value proposition, overall length, complexity and flow must all be considered, while still addressing the key research objectives. This is not a balancing act for the untrained to tackle.

3. You haven’t thought through how the data will be analyzed.

It’s not enough to brainstorm a list of questions for which the responses will be “interesting.” Too often, that leaves novices with a file full of data they’re not sure how to use to their best advantage. A good researcher knows how to think backward from the objectives. It’s a specialized skill to be able to understand the analyses needed to accurately address the objectives, then envision the structure and type of data that will be required to perform those analyses. From there, an experienced researcher can design a questionnaire that will accurately and effectively populate the required data fields and successfully fuel insightful analysis.

4. You don’t know why some formats are better for certain questions.

Related to #3 above, there are usually several possible ways to ask any given research question. Consider the following examples:

1. What sporting events do you plan to attend this month? (open-ended)
2. Are you going to attend a sporting event in this month? (yes/no/not sure)
3. How many sporting events will you attend in the next 30 days? (numeric response)
4. How many of each of the following types of sporting events will you attend in the next year? (grid with ranged responses)

It’s important to consider how the data will be used before settling on question structure. I’ve had clients write up questions in all of these formats but most are unaware of the proper uses, benefits and disadvantages of any particular one.

Here’s a pop quiz on just this one example among dozens of factors requiring evaluation for every single survey item. Match the question formats above with the advantages and/or disadvantages listed below:

a. Often used to screen respondents for qualification; not useful for deep segmentation (Answer: Q. 2)

b. Used to evaluate relative standing; space-efficient but adds to respondent fatigue and usually requires rotation or randomization (Answer: Q. 4)

c. Good for exploratory insights; creates respondent fatigue and often requires additional back-end coding (Answer: Q. 1)

d. Allows frequency-based analysis and segmentation; prone to overestimation (Answer: Q.3)

5. You don’t suffer over every word.

Okay, maybe “suffer” is too strong a word but a talented survey writer can step back from a survey item and look at it from all angles. Does it really ask what you think it asks or is there room for multiple interpretations? Is there wording that creates subtle or implicit bias? Is the question, as crafted, likely to lead respondents to answer more positively or negatively? What are the impacts of previous questions in the same survey? Do the response codes and options align to the question and are they written in a way that allows every respondent to provide an accurate and complete answer? Each of those questions (and many others) is the subject of literally hundreds of research community articles, debates and experiments. It’s simply not possible for someone with little to no knowledge of this background to adequately weigh these factors. Trust a trained research professional to help you create survey questions that will give you accurate, actionable and projectable data.

6. You don’t have time and money to burn.

It costs more, and takes more effort, to clean up or revise a poorly-written survey than to write a good one from scratch. There is no value to surveys which, through inexperience, deliver incomplete, unactionable or inaccurate data. I can’t tell you how many times I’ve had a client insist they will “save money” by writing their own survey, only for them to end up with unforeseen expenses and extended timelines when the results are suboptimal at best or unusable at worst. As professional researchers, we get it: We know your budget is limited and your timeline is tight. We want to help, really! When it comes to survey research, the GIGO principle (garbage in, garbage out) cannot be dismissed. Don’t be afraid to invest a few hours of your research professional’s time on upfront survey design – it will pay off in data accuracy, quality and utility.

Your dentist genuinely loves teeth and has spent years studying and practicing the best ways to take care of them. A trained research professional genuinely loves survey design and analysis and has invested significant time and effort in honing those skills. Let us keep you, and your business, all smiles.