Are hyperactive respondents different?

Editor’s note: Steven Gittelman is president, and Elaine Trimarchi is executive vice president, of Mktg, Inc., an East Islip, N.Y., research firm.

Hyperactive respondents participate in multiple online panels, take many surveys and in many cases have been panel members for years. These professional survey takers complete a disproportionate percentage of online surveys. The sheer number of interviews they complete represents a response bias, but if they are behaviorally different from less-active respondents and the population in general, they will degrade the representativeness of online research. While there has been a great deal of literature disputing this point, we will show that these participants appear to be very different and their overabundance in panels represents a threat to data quality. We believe that only by monitoring these effects and tracking the participation of hyperactive respondents can the quality and consistency of these panels be established and maintained.

Sample attrition in online panels may cause differential sifting with the end result being that the remaining respondents differ from those who begin the process.

Background

Chang and Krosnick conclude that a study of traditional panels (e.g. Fitzgerald, Gottschalk, and Moffitt 1998a, 1998b; Falaris and Peters 1998; Zagorsky and Rhoton 1999; Clinton 2001) showed little or no sample change attributable to panel attrition. They add that it is likely to be even “less pronounced on Internet panels covering diverse topics over time.” All of these papers, with the exception of Clinton, draw their conclusions from demographics. More recently a report on online panels prepared by AAPOR (Baker, et. al., 2010) found that little research existed on the subject and suggested that the commercial panels have the data and should present the research.

We hypothesize that the panel member who survives over time is likely to represent a particular set of behavioral characteristics. It is counterintuitive to conclude that persons who are willing to participate in 30 or more surveys per month, belong to greater than five online panels and continue to offer such dedication to the process for two or more years could behave similarly to new panel members, who are likely to drop out of the process entirely, complete very few surveys and never join more than one panel.

The idea that some people are more inclined than others to take surveys is, of course, not new and it is not simply a result of the use of online methods. Bickart and Schmittlein (1999) estimated that between 20 percent and 23 percent of U.S. adults accounted for all survey responses using any data collection method and that just 4 percent to 5 percent of adults accounted for more than half the total survey responses. The increasing use of online panels has almost certainly concentrated the majority of survey taking within an even smaller sliver of the population. In 2005, ComScore Networks estimated that only 0.25 percent of all Internet households accounted for more than 30 percent of all online surveys taken (Fulgoni, 2005). Recent data, reported by the Advertising Research Foundation (Walker, Pettit and Rubinson, 2009) has indicated that the average online respondent is an active participant in 4.1 online panels. Gittelman and Trimarchi (2009) reported that respondents belong to an average of 4.4 online panels and frequently stay on those panels for years. Furthermore, some respondents complete dozens of surveys every month. These are likely to be different from new, less-experienced respondents.

In fact, according to a new study by North Carolina State University on twins, there is a genetic component to whether an individual takes surveys. In a comparison of identical twins, who share the exact same genetic makeup, and fraternal twins, who share only half their genes (like normal siblings) it was found that a strong correlation in survey-taking existed between identical twins that did not exist between fraternal twins. Fraternal twins share almost identical (except gender) demographic characteristics - it is difficult to think of a more compelling argument that demographic quotas are no longer sufficient to act as a sample frame standard.

The existence of hyperactive respondents is not at issue, today. However, the potential impact of these respondents on the results of surveys is a key topic. Major attitudinal differences were found in respondents who belonged to multiple panels (Casdas, Fine and Menictas, 2006) although they were demographically similar to those in only one panel. The differences appeared to go far beyond demographics and it was suggested that they required weighting by covariates. “Panel conditioning” has been described (Chang and Krosnick, 2009) as a “potential drawback of repeated interviewing, …whereby accumulating experience at doing surveys makes panel members less and less like the general public they are intended to represent.” In a number of studies, the effect has been found to be minimal or non-existent (e.g., Himmelfarb and Norris, 1987; Cordell and Rahmel, 1962; Sobol, 1959). Willson and Putnam (1982) warned that, “There is a general pretest effect which cannot be ignored.” In some cases, they suspected systematic bias, particularly in non-random sampling frames. Further, they found that the effects were inconsistent and thus found it difficult to provide definitive solutions. Chang and Krosnick (2009) found that practice effects altered quality, deducing that the response behavior of panelists is influenced both by practice and interest in subject matter, tying their work into a long tradition of supporting literature on both subjects.

Panel tenure has been identified as a cause of poor predictive validity. Ron Gailey examined why more than a dozen studies that his company conducted showed unrealized decreasing demand for products. He concluded that those research results were fatally affected by a shift in respondent panel tenure that changed the participation of professional respondents (Gailey, 2008).

Inexperienced respondents were more favorable on brand purchasing intent questions than experienced respondents (Coen, Lorch, Piekarski, 2005). Completion of multiple surveys was also found to be a more sensitive measure than tenure. Panelists who participated in a greater number of questionnaires were more negative to a concept while those who had been long-term but less active members of a panel did not share the same shift of opinion. The authors concluded: “This research underscores the importance of maintaining a stable panel and of using a panel which can provide a consistent mix of respondent experience over time.” Coen et. al. concluded that “intent to buy” responses given by frequent survey takers were more in line with reality. A similar result was reported by the Advertising Research Foundation (Walker, et. al. 2009).

Walker, Pettit and Rubinson, working with 17 American panels and a sample size of some 100,000 respondents, declared that the panels were not interchangeable. The driver(s) of this variance appeared to be multivariate. The authors discount multiple panel membership and frequency of survey taking as drivers of variance. The thrust of their analysis on purchasing intent was through soup and paint choice tests, where no measure of hyperactivity appeared to drive changes in purchasing intent. They concluded: “The data suggest that panel practices work together in subtle ways to build groups of respondents with distinctive attitudinal profiles. Though panel tenure may be one such factor, the way panels recruit, the type and amount of incentives offered, and possibly even the ‘character’ of an individual research/panel company may encourage distinctive panels to emerge whose members share attitudinal and motivational propensities that drive results that may vary from panel to panel.”

Procedures

Meaningful difference in the characteristics between groups of respondents is no guarantee that survey results of specific questions will be different. Some questions are hyper-stable and do not vary with the differences in these groups. Unfortunately, we usually cannot tell when this will happen. What we need are metrics that reflect the fundamental consistency and commonality of samples. Here we test new measures to detect variability in online panels. These metrics are based on the distribution of structural segments, which are derived from the statistical analysis of buyer behavior, sociographic and media attitude questions from a standardized questionnaire. These metrics are sensitive measures of differences between samples and have been used here to examine the potential impact of professionalism. They are robust measures based on over 30 variables each and are therefore less sensitive to individual responses, thus reflecting the characteristics of the sample.

The current analysis involves 18,561 responses to a Web survey provided by individuals from 17 different U.S. online sample sources. Respondents were collected from May 2008 through February 2010. Quotas were employed for age, income, gender and ethnicity to reflect the census. No effort was made to monitor sample recovery. As these are access panels, sample recovery can be manipulated by choosing the most reliable frequent responders. Given the propensity of certain respondent groups to participate in online surveys, it appears that response rate can easily be managed by choosing among frequent participants. However, there is an inherent risk in such practice: If the frequent responders are different from the population, bias is being injected into the sampling arena.

Figure 1 shows an illustration of the potential impact of professionalism on survey results. The chart shows greater than a twofold change in the frequency of buyer behavior segments between those respondents who have not been on a panel more than six months and those who have been on panels for more than two years. As you will see, this is a typical characteristic and is reminiscent of data reported by Gailey, 2008, where there are dramatic shifts in purchasing intent that progress through time.

Individuals completing the questionnaire were placed into one of four buyer behavior segments based on responses to 37 input variables. Almost half of the respondents who report taking three or fewer surveys per month are placed within a buyer segment that tends to shop on credit rather than waiting. On the other hand, 20 percent represent individuals who report taking many surveys (more than 30 per month) are disproportionately placed in a segment that skews high on use of coupons (Figure 2).

Similar results are seen when subdividing respondents by the number of panels to which they report belonging. On average, these 18,561 respondents - most derived from well-known online sample providers - indicate being members of 4.4 panels. Almost 45 percent acknowledge being a member of at least five panels. On the other hand, 23 percent of these respondents say they are members of no panels at all.

Almost half of those who claim to be in no more than one panel fell into the buyer behavior segment called “Broad Range/Credit.” This segment is composed of people who report relatively high levels of behavior on virtually half of the items used for the segmentation. These items include: making high-tech purchases, downloading music, listening to Internet radio and buying on credit rather than waiting. Members of eight or more panels fall disproportionately into the “Domestic/Coupons” segment. This segment is composed of people who tend to report relatively low levels of behavior on these measures except when it comes to buying American-made products and using coupons   (Figure 3).

Panel tenure also affects how respondents are distributed among the buyer behavior segments. Half of the respondents who indicate being a member of a panel for less than six months fall into the “Broad Range/Credit” segment, while those that are veteran members and indicate 24 or more months of service in at least one panel disproportionately fall into the “Domestic/Coupons” segment. Distributions into the other two buyer behavior segments - “Price-Sensitive Shoppers” and “Credit/Environment” - vary, but less dramatically across these measures (Figure 4).

Media attitudes in question

In addition to looking at differences in buyer behavior, survey-taking experience was analyzed in the media usage segments. Some might assume that those taking more surveys and joining more panels would be computer-oriented respondents. Figure 5 suggests that it is those who take fewer surveys and have joined fewer panels for shorter periods of time that fall heavily in an Internet media segment in which activities such as social networking, video downloading and instant messaging tend to be high.

This imbalance suggests that, if a media or channel decision were to be based on results of an online survey, the recommended decision could vary greatly depending upon the blend of tenure and hyperactivity represented by respondents who answer the survey. If they were less-tenured online respondents who tend to join fewer panels and take fewer surveys, a higher emphasis could be placed on online advertising than if they were online survey-taking veterans.

Tempted to blame the respondents

While researchers might be tempted to blame the respondents who take so many surveys for these difficulties, it is hard to say it is really their fault. The online research industry has developed in such a way that many companies have economically benefited by having such people join panels and self-select their way into surveys by deciding to respond to invitations sent to them. Some researchers and suppliers have promoted the concept of achieving online respondent quality by using carefully prepared sub-populations. These resulting panels are then heavily used, resulting in an increase in professional participation. 

The reality is, respondents provided by online sample sources are going to contain a mix of short- and long-tenured panelists - neither set is readily removable. Organizations supplying respondents to online surveys for the most part depend upon a business model that allows them to recoup recruiting costs by using their panelists multiple times. While most will continually recruit in order to keep their panel numbers up, continually supplying surveys with only “fresh” respondents would result in a massive disruption of that business model, even if getting a less-involved set of respondents were to be deemed positive.

Provide improved quality

Filtering or culling potential “bad” respondents has been proposed as a means to provide improved quality of online panels. This is applied in any number of ways to reduce problem respondents and to force the sample to be in agreement with specified demographics. In many cases, additional culling is necessary; for example, to remove duplicate respondents and clearly erroneous behavior. Some efforts are being made to cull “the worst” respondents from online sample pools.

However, rarely is culling used to restrict hyperactive survey takers from samples. The issue of hyperactive respondents is usually not associated with sampling frame filtering. Quota controls based on demography and/or sample frame selection by probabilistic methods matter little if the individuals retained represent the demography of a population but are substantially different in their behavior.

Clearly, data source characteristics, including the incidence of professionals, vary among panels within a country. Within-country variability in the sampling frame is increased by the merger of suppliers and the ebb and flow of convenience partnerships created when single companies cannot reach quotas and need to reach beyond the ability of their own resources to fulfill them.

Blend data sources

An alternative method to stabilize the characteristics of respondents is to blend data sources. We suggest that segmentations such as those used above are broad measures of behavior that could be used as blending targets. To do so, baseline data must be collected to generate combinations of panels that approximate the collective Web survey population within a country or market. This method would then employ various optimization models to create recommended blends of panels that most closely match this overall grand mean. The resulting blends should be stable at least to the extent that they are more representative of the total Web survey populations.

We conclude that testing online populations against single measures is unwise. We risk choosing those variables that are hyper-stable. Instead, we support a multivariate approach.

As important as the performance of individual data sources is, their consistency is critical. Walker et. al. conclude that online panels are consistent through time. Their test spanned a matter of only a few weeks. There are many potential reasons why an online sample source might be inconsistent across time. The changing presence of professional respondents is clearly one of the most important. Clients using online sample sources should insist that the pool of individuals surveyed this month be similar to those surveyed in the future. Without such assurances and a means for monitoring them, the casualty will be the survey results themselves. 

References

Bickart, Barbara and David Schmittlein. 1999. “The Distribution of Survey Contact and Participation in America: Constructing a Survey-Based Estimate,” Journal of Marketing Research. 36:(2):286-294.

Bridge, R. Gary, Leo G. Reeder, David Kanouse, Donald R. Kinder, Vivian T. Nagy, and Charles Judd. 1977. “Interviewing Changes Attitudes - Sometimes” Public Opinion Quarterly 41:57-64.

Casdas, Dimitrio; Brian Fine, Con Menictas. 2006. “Attitudinal Differences: Comparing People Who Belong to Multiple Versus Single Panels.” Panel Research 2006. ESOMAR World Research Conference.

Chang, Linchiat; and Jon A. Krosnick. 2009. “National Surveys via RDD Telephone Interviewing versus the Internet Comparing Sample Representativeness and Response Quality.” Public Opinion Quarterly 73(4)641-78.

Clausen, Aage R. 1968. “Response Validity: Vote Report.” Public Opinion Quarterly 32:588-606.

Coen, Terrence; Jacqueline Lorch, Linda Piekarski. 2005. “The Effects of Survey Frequency on Panelists’ Responses.” In: ESOMAR World Research Conference, Developments and Progress, Budapest, 17-19 April 2005, p. 409-424.

Cordell, Warren N., and Henry A. Rahmel. 1962. “Are Nielsen Ratings Affected by Non-cooperation, Conditioning, or Response Error?” Journal of Advertising Research 2:45-49.

De Wulf, Kristof; Sam Berteloot. 2007. “Duplication and Multi-Source Online Panel Recruitment: Real Quality Differences or Idle Rumours?” ESOMAR World Research Conference, Panel Research 2007: Good Quality, Good Business, Orlando, 28-30 October 2007, 49-62.

Falaris, Evangelos M., and H. Elizabeth Peters. 1998. “Survey Attrition and Schooling Choices.” The Journal of Human Resources 33:531-54.

Fitzgerald, John, Peter Gottschalk, and Robert Moffit. 1998a. “An Analysis of Sample Attrition in Panel Data: The Michigan Panel Study of Income Dynamics.” NBER Technical Working Papers, National Bureau of Economic Research, Inc.

_____. 1998b. “An Analysis of the Impact of Sample Attrition on the Second Generation of Respondents in the Michigan Panel Study of Income Dynamics.” The Journal of Human Resources 33:300-344.

Foster Thompson, Lori. 2010 “Survey Says: Genetics Affect Whether We’re willing To Take Surveys”. North Carolina State University.

Fulgoni, Gian. 2005. “The ‘Professional Respondent Problem in Online Survey Panels Today,” presentation at Market Research Association Annual Conference, Chicago, June 2005.

Gailey, Ron. 2008. “The Quest for Sample Integrity: Exploring Panel Sample Characteristics That Impact Survey Results,” presentation at The Research Industry Summit: Solutions that Deliver Quality, IIR, Chicago, November 2008.

Gittelman, Steven and Elaine Trimarchi. 2009. “On the Road to Clarity: Differences in Sample Sources.” CASRO Panel Conference. New Orleans. 2009.

Granberg, Donald, and Soren Holmberg. 1991. “Self-Reported Turnout and Voter Validation.” American Journal of Political Science 35:448-59.

Greenwald, Anthony G., Catherine G. Carnot, Rebecca Beach, and Barbara Young. 1987. “Increasing Voting Behavior by Asking People if They Expect to Vote.” Journal of Applied Psychology 72:315-8.

Himmelfarb, Samuel, and Fran H. Norris. 1987. “An Examination of Testing Effects in a Panel Study of Older Persons.” Personality and Social Psychology Bulletin 13:188-209.

Huizing, Lennart; Robert van Ossenbruggen, Martin Muller, Clasine van der Wal, Gerty J.L.M. Lensvelt-Mulders and Marianne Hubregtse. 2007. “Improving Panel Sampling: Embedding Propensity Scores and Response Behavior in Sampling Frames.” ESOMAR World Research Conference, Panel Research 2007: Good Quality, Good Business. Orlando, 28-30 October 2007, p. 108-123.

Mann, Christopher B. 2005. “Unintentional Voter Mobilization: Does Participation in Pre-election Surveys Increase Voter Turnout?” Annals of the American Academy of Political and Social Science 601:155-68.

Pineau, Vicki and Daniel Slotwiner. 2003. “Probability Samples vs. Volunteer Respondents in Internet Research: Defining Potential Effects on Data and Decision-Making in Marketing Applications,” Knowledge Networks Inc.

Sherman, Steven J. 1980. “On the Self-Erasing Nature of Errors of Prediction.” Journal of Personality and Social Psychology 39:211-21.

Sobol, Marion G. 1959. “Panel Mortality and Panel Bias.” Journal of the American Statistical Association 54(285):52-68.

Sparrow, Nick. 2007. “Quality Issues in Online Research.” Journal of Advertising Research June 2007.

Taylor, Eric. 2007. “Are Panelists Employees or Respondents? Social Versus Economic Exchange.” in: ESOMAR World Research Conference, Panel Research 2007: Good Quality, Good Business. Orlando, 28-30 October 2007, p. 19-32.

Walker, Robert, Raymond Pettit, and Joel Rubinson. 2009. “A Special Report from The Advertising Research Foundation: The Foundations of Quality Initiative, A Five-Part Immersion into the Quality of Online Research.” Journal of Advertising Research 49: 464-485.

Willson, Victor L., and Richard R. Putnam. 1982. “A Meta-Analysis of Pretest Sensitization Effects I Experimental Design.” American Educational Research Journal 19:249-58.

Zagorsky, Jay and Pat Rhoton. 1999. “Attrition and the National Longitudinal Survey’s Women Cohorts.” Columbus, Ohio: Center for Human Resource Research, Ohio State University.