Sex, lies and the Internet
Editor’s note: Michael C. Sack is president of Image Engineering, Inc., a Goshen, Ky., research firm.
A few decades ago, a couple of researchers named Masters and Johnson began researching an aspect of human behavior that is considered both very important and very personal. They asked men and women straightforward questions about sex.
When they examined responses by gender, Masters and Johnson noticed that men reported much more frequent sexual activity than did women. The results suggested only two possible conclusions - the men either weren’t supplying straight answers, or else they weren’t engaging in “straight” behavior!
The continuing challenge for researchers in the field of human sexual behavior is to minimize lying and to quantify the “liar factor,” so researchers can estimate actual behavior.
What does this have to do with the Internet? Everything. We’re talking about the medium that gave rise to chat-room alter egos and e-mail urban legends. Falsehood and pretense are as much a part of the online fabric as “www.” So, before we hail the Internet as the greatest advance for researchers since statistics, we need to learn how to minimize lying and to find better ways to estimate actual behavior.
My recent research shows that respondent lying about Internet activity may be far greater than most people think. It not only applies to market research collected on the Internet - it pertains to market research that has anything to do with the Internet. Why? Because Internet activity, like sexual activity, is private behavior that deeply impacts one’s sense of personal significance. Because the subject is considered important, respondents strongly desire to see themselves (and to be seen by others) in the best possible light. This makes self-reported information highly susceptible to misrepresentation.
The case of the lying respondents
An example of this tendency, and some ideas about what can be done to counter it, can be seen in the case study of awareness-and-usage research for an e-business begun by an established, catalog-based health-and-wellness company.
The company commissioned a tracking study to examine the impact of advertising on awareness and use of its Web site. The target market was women, ages 30-60, who use the Internet at least one hour a week, who have searched for health-related information on the Internet, and who have positive health-related attitudes and behaviors.
In the first wave of research, fielded in September 1999, several firms were used to recruit more than 500 participants by phone for a central-location study fielded at 10 sites nationwide. In addition to answering qualifying questions, recruits also passed a standard security screen to ensure that they and their family members were not employed in jobs related to health, medicine, marketing research, journalism or the Internet.
Qualified participants were invited to come to the research facility, where they were re-screened before admission to the study, which included group discussion and a computer-aided interview.
At the research facility the first liar factor surfaced right away. More than 10 of these “regular Internet users” didn’t know how to use a computer mouse. Another 70 demonstrated insufficient computer skills to take a computer-aided interview.
At the beginning of the computer-aided interview, the security questions were repeated again. This time another approximately 80 participants altered the answers they had given twice previously during screening. Most often this “misreport” was about medical professionals in the family.
At this point, the usable pool of recruits had dropped from more than 500 to 355. But the misrepresentation continued. During the computer interview, a third of respondents reported spending 30 minutes or less each week on the Internet (vs. the hour-per-week minimum required to qualify for the study). Overall, of 355 respondents, 235 provided at least one set of inconsistent responses.
During the in-person interviews which included group discussion, the moderator noted that participants’ comments were “punctuated by hyperbole.” Respondents also expressed themselves with two different vocabularies. When speaking generally about the Internet they used a “social” vocabulary derived from the media. But when they discussed their own Internet usage or evaluated specific Web sites, they used a separate, “private” vocabulary devoid of industry jargon. Terms like “navigation,” “portal” and “load speed” disappeared.
A second wave of this same research was also conducted. This time, recruiters screened potential participants with stricter, more focused questions about computer usage and skills. Still, about 20 percent of qualified respondents were lost for security or insufficient skills (down from 33 percent in Wave 1).
During the second wave, the “outing” of those with insufficient skills took a humorous turn. Participants made excuses to leave the room — and never returned! Internal inconsistencies declined, but they were still found in 50 percent of responses (down from 66 percent in Wave 1).
What the lying suggests about consumers
These results provide a useful perspective on consumer values. Most people welcome technological advances, but those advances outpace people’s ability to keep up, in terms of both time and finances. Nevertheless, people desperately want to view themselves as technologically up-to-date. To see themselves otherwise would be to succumb to “techno-aging,” public acknowledgment that the world has passed them by. Psychologically, this acknowledgment can be considered a form of social “death.”
So, consumers do their best to keep up, and do even better at convincing others that they keep up. Because the behavior is private, creating a tech-literate persona is easy. All it takes is the ear to develop a limited technical vocabulary (such as the one selectively practiced by respondents during the group discussions).
When people exercise a persona long enough, they can easily convince themselves that their public face is their true identity. This can lead to an unwitting overestimation of skills and experience.
But there’s another reason people might misrepresent themselves to survey researchers: an easy opportunity to acquire skills or to have a technology-related experience. It’s a learning opportunity where the student actually gets paid to participate!
What the lying suggests to researchers
The case study reveals several valuable points for researchers. First, know the susceptible categories. Researchers should show special care in dealing with products and services that are privately consumed, yet closely linked to self-image. When consumption is usually not observed, consumers are easily able to create an image for others.
Second, be as specific as possible in screening. Focusing on details in screening not only provides better information, but it also communicates that the researcher considers accuracy to be important.
Third, don’t be afraid to repeat important questions. If we hadn’t asked questions three times, we wouldn’t have had some substantial security breaches. When repeating questions, varying the data collection method used can be helpful. It can also be helpful to re-screen after respondents believe they have successfully bypassed the screen. In our case study, several may not have expected the computer to “catch” their disqualifying response.
Fourth, use internal consistency checks. Often, researchers set up surveys to disallow inconsistent responses. Instead, consider allowing inconsistent responses, building in a mechanism to clarify responses. This preserves your clean data while also providing a measure of the degree of inconsistency of each respondent. When a respondent provides too many sets of inconsistencies, drop him or her from the data set.
Fifth, plan for the liar factor. Gather more sample than you need, and be willing to invest in the process of weeding out those who misrepresent themselves. If you don’t feel the freedom to throw out bad responses, you’re likely to be an accessory to the crime.
Sixth, be careful when using socially-oriented forms of data collection. Phone interviews and personal interviews can be great for the right applications, but when respondents are concerned about how others perceive their answers, it may be better to ask via computer or paper-and-pencil. The more truthful responses in our case study came via computer. (This actually bodes well for Internet surveys!)
Seventh, monitor interviews and use personal observation whenever possible. By using computer-aided interviewing in a central-location study, we were able to confirm that self-described Internet users really knew how to use the Internet.
Eighth, use non-traditional data collection methods to bypass defenses. Many techniques require researchers to trust the veracity of respondents. Projective techniques bypass conscious defenses to access information that respondents may not be aware of. I often collect data using visual images, which contain symbolism that goes far beyond respondents’ conscious reasoning.
Ninth, use tracking. No matter how careful you are, you are still going to have some misrepresentation coming through. While measuring actual activity can be difficult in such an environment, measuring change in activity is much easier. If the liar factor remains relatively consistent from wave to wave, incremental changes should be relatively accurate measures.
A final warning
If you read through the above case study saying to yourself, “My Internet respondents aren’t like that; I can trust them,” you are especially susceptible to the liar factor. Give your respondents the opportunity to lie (and be caught) and see how they do. Then arm yourself with tools to minimize misrepresentation. In the end, you will gain a better understanding of your consumers - whether you’re dealing with Internet research, sexual behavior research, or any product category that’s consumed in private.