Editor's note: Joe Hopper is president of Chicago-based Versta Research. This is an edited version of a post that originally appeared under the title, “Decrease survey error with eye-tracking diagnostics.”
If you make your survey questions hard, it turns out that people will make a good faith effort to answer them. But they will make mistakes and you will end up with bad data that lacks reliability and validity.
What I find so interesting is that most people will actually try to answer your hard questions. Unless they’re plain-old cheaters they do not speed through and provide random answers to your bad questions. They actually attempt to sort through and untangle them.
Evidence for this can be found in a study from a team of researchers at Penn State, Ghent University and Tilburg University published in The Journal of Marketing Research. They looked at what happens when statements like this are presented to respondents with negations:
- “When I travel I don’t like taking a lot of photographs.”
- “When I travel I don’t hate taking a lot of photographs.”
- “I am not the kind of person who would try any new product once.”
- “I am not the kind of person who would rather stick with a familiar product.”
Those are hard statements to get your head around. But the researchers found it was not necessarily respondent speed and inattention that generated errors in giving answers when they were presented with confusing statements. It was lack of comprehension. How do they know? The researchers analyzed eye-tracking data and time data as people filled out the survey. They found that respondents took more time on these statements and that respondents were inordinately focused on the complicated negated phrases.
They also showed that statistical analysis, using factor analysis, for example, was not sufficient to identify the weaknesses in these statements and the errors caused (in their case, they were using items for scale construction). Eye-tracking data was crucial to pointing out the source of difficulty.
Of course I hope you never write statements as difficult and convoluted as those shown above. The authors offer a recommendation to avoid “negated polar opposites” and other complicated forms of negation in writing survey questions. You don’t need eye-tracking technology to develop a good feel for simple use of language.
But if the project you are designing is going to involve a lot of time and money (large samples, longitudinal research, etc.), eye-tracking may give you deep insight into trouble spots where people are struggling to comprehend your questions and what may be required of them.