Reducing risk
Editor’s note: Michael A. Fallig is senior vice president, online strategy and methodology division at GfK Research Center for Excellence, New York. Derek Allen is executive vice president, director, GfK Research Center for Excellence.
Online sample for survey research can be sourced from a variety of providers, most offering samples from non-probability respondent pools. Resources include: consumer access panels with members who were recruited to take surveys; online communities that have members who earn points for engaging in a variety of activities, including taking surveys, but were never expressly recruited to take surveys; providers who have access to members who joined social communities and social network sites; site visitors who are willing to take surveys when an invitation is available; lists, databases, Web site intercepts, blogs and other resources.
Many online research programs could benefit from drawing project sample from several online respondent pools rather than just one. If done properly, sample from multiple pools can:
• broaden coverage of the characteristics of the population at large;
• meet the needs of studies with unusually demanding characteristics (particularly continuous tracking studies and market-level studies):
-- extremely low incidence of eligibility
-- large sample size requirements
-- very localized geographic requirements
-- lengthy past participation or study lockout requirements
• reduce risk of reporting or using an out-of-range response measure estimate for important decision-making.
The intrinsic and extrinsic differences that define who we are as individuals are considerably more plentiful in variety and number than the handful of demographics that some sample providers try to manage via sample deployment distributions, quotas or weighting. These individual differences may sometimes be more strongly associated with key response measures than demographics. In such cases, if each non-probability respondent pool was comprised of a different mix of personal characteristics, then it would stand to reason that each pool would report a different response measure distribution.
We hypothesize that the mixtures of personal characteristics within each non-probability respondent pool will be different, in part because of the disparity in the methods used by respondent-pool providers for attracting, recruiting, engaging, maintaining, incentivizing and replacing members and visitors. Each provider’s set of methods may appeal to, keep engaged and tenured, qualitatively different mixtures of people.
Furthermore, each community and “river” content site provides its own unique experience to its members or visitors, including its own mix of subject matter, activities, membership interaction opportunities; and the number, frequency and type of surveys its members and visitors participate in, etc. Therefore, even if providers were able to attract or recruit the same types of people initially, these pools would become qualitatively different over time as each pool obtains more and more unique exposure experiences as members and site visitors interact with and build tenure with the communities and content sites they frequent.
Constellations of characteristics
Because most online respondent pools are not probability samples of either the adult U.S. population or the adult Internet population, no single respondent pool is expected to reflect the U.S. or U.S. Internet population at large. Some pools may be light on certain constellations of these characteristics (which we will outline below), such as being open to change, and heavier than the population at large on others, such as being interested in volunteering.
We hypothesize that these differences, intrinsic and extrinsic in nature, contribute importantly to the differences observed in some response-measure distributions observed across samples from different respondent pools. These response-measure differences remain after fraudulent respondents, satisficers and the like have been removed from each sample and the remaining members are balanced on key demographics.
A challenge
Creating a recipe for properly drawing sample from multiple resources is a challenge that anyone interested in blending is faced with today. Some have taken an arbitrary approach, fixing the percentage contribution of each resource. However, we are not aware of any scientific approach for determining the proper blend of resources.
Because it is not easy to predict when a particular non-probability finite respondent pool will return an out-of-range result, GfK has been working on a solution to increase the comfort level with using these pools. Our approach does not focus on blending resources per se but on the following broad premises:
1. Attraction, recruitment, engagement, incentive, replacement and community or content site characteristics play roles in the type of people who are attracted to, sign up with, maintain membership/affinity with, or repeatedly visit survey and social communities and content sites.
2. As a function of the variety of approaches and offerings of providers, communities and content sites recruit and maintain people who differ intrinsically and extrinsically.
-- Extrinsic characteristics that are likely to be related to survey responses are also likely to be less universally exhibited across the population at large, and are not always easy to identify or control: panel tenure, survey participation, Internet activities and so on.
-- Intrinsic characteristics of importance are stable and more easily controlled or accounted for: personality traits, values, locus of control, need for cognition and functional attitudes.
3. These same internal characteristics appear related to issues such as:
-- Willingness to volunteer, donate or comply with a request such as join a community, take a survey, etc.
-- Breadth of online and offline activities people engage in including the time they spend online.
4. A major source of situational extrinsic differences is a function of the community and site activities that members and visitors engage in - and the surveys that they participate in and the subject matter they commit to long-term memory.
5. Given the assumed properties of the intrinsic and extrinsic characteristics, a prudent strategy may include controlling the distribution of important intrinsic characteristics sampled while randomizing as best as possible the myriad of extrinsic characteristics that could impact results (by randomly selecting people from a number of diverse respondent pools).
6. Online respondent pools are comprised of several different latent classes of people. Each class is identifiable by a set of intrinsic characteristics in addition to demographics:
-- online respondent pools will consist of the same set of latent classes;
-- the size of each class and the distribution of the classes within each respondent pool are likely to be different.
7. Non-probability online samples can benefit by calibrating the latent class sizes to the size of the classes found within the Internet population at large as measured using an instrument administered offline to a gold standard probability sample (e.g., area probability sample, ABS, etc.).
Enhance the quality
Today, many sample providers, service bureaus and full-service research firms are engaging in procedures to enhance the quality of their online respondent pools and/or study samples. The current efforts appear focused on eliminating frauds, duplicates, satisficers and the like. An underlying assumption is that with these issues eliminated, differences across online samples will be reduced and response patterns will be more believable.
We do not believe that the current focus diminishes the myriad of individual differences found among members of a respondent pool and across different pools of respondents. We believe that these individual differences need to either be carefully controlled or randomly distributed within each sample that is drawn.
Individual differences can be characterized as intrinsic or extrinsic in nature. Among intrinsic differences, we include personality characteristics, values, locus of control, demographics and other long-lasting and stable personal characteristics. Extrinsic differences include panel tenure, number of surveys people have taken, frequency of dining out, attending a ballgame, visiting the theater, blogging, shopping habits, number of hours spent online, among other things.
We hypothesize those providers’ differences in how they attract, recruit, maintain, engage, incentivize and replace their respondent pools play roles in why respondent pools are different from each other. Considering that strategies are also used for maintaining affinity with social community members and visitors to content provider “river” sites, survey respondents obtained from all these resources are likely to develop their own distributions of individual characteristics.
We further hypothesize that the distribution of intrinsic and extrinsic individual differences in respondent pools are not the same and contribute to the response measure differences observed across online respondent pools even after resources are equated on standard demographics by weighting or balancing.
We have also assumed that a finite number of important intrinsic factors exist that are important to control but there are far too many extrinsic factors to identify or control. Consequently, extrinsic factors should be randomized as best as possible. If randomization is not feasible, then their influence should be reduced by drawing sample from a variety of respondent pools.
Considered to be universal
In developing our model, we took care to examine intrinsic personal characteristics that we considered to be universal across cultures and countries and relatively stable. We also took care to stay away from developing a model that would optimize on a specific response measure, such as purchase propensity. It was felt that such an approach would reduce the ability to use the model across all types of online studies. However, if the goal is to optimize sampling for a specific area, such as election polling, then consider a different dependent variable and mix of independent variables.
Research has been sparse with regard to the exploration of individuals’ intrinsic characteristics and their association with survey participation, consumer panel membership, Internet use and the like.
Research regarding compliance with requests, altruism, volunteering and other pro-social and helpful behaviors reveals a positive association with certain personality traits and dimensions. For example, agreeableness, openness and extraversion - Costa, McCrae, Dye (1991); Jang and Livesley (1996); Bekkers (2004); Glendon, McKenna, Clarke (2006) - are found to be related to volunteering and other pro-social behaviors, including volunteering to take surveys.
Studies examining the relationship between personality traits (Big Five Inventory) and personal values (Schwartz Values Survey) reveal robust associations among clusters of values and specific personality traits. For example, agreeableness appears to be associated with traditional values and openness with universalism (Roccas, Sagiv, Schwartz and Knafo, 2002). The connection between personal attitudes and values has also been explored as attitudes often emerge by means of cognition from personal core values.
An extensive review of the literature was conducted to investigate intrinsic, extrinsic and motivational factors that were related to the following areas of concern:
• Internet usage and Internet activities;
• willingness, compliance and reluctance to participate in surveys and in research;
• attitudes toward surveys and taking surveys;
• motivations for joining online panels, social and virtual communities;
• motives for participating in online surveys and as an active member of an online social community;
• volunteerism; and
• decision-making (e.g., shopping, purchasing, political, etc.).
After careful consideration, GfK included the following measures in its intrinsic model as independent variables:
Big Five personality traits (BFI)
The Big Five Inventory is based on the works of Cattell (1943), Norman (1967) and Goldberg (1990). The 44-item battery yields five personality factors: neuroticism, openness to experience, extraversion, agreeableness and conscientiousness.
Portrait Value Questionnaire (PVQ)
The PVQ, developed by Shalom Schwartz (2001) and based on value theory, measures 10 basic overarching values: conformity, tradition, benevolence, universalism, self-direction, stimulation, achievement, power, security and hedonism.
Need for Cognition Scale (NCS)
The 18-item Need for Cognition Scale was developed by Cacioppo, Petty and Kao (1984), and measures individual preferences and tendencies to engage in complex thought, by exploring the motivational aspects of information processing.
Consumer Locus of Control Scale (CLOC)
The Consumer Locus of Control Scale developed by Busseri, Lefcourt and Kerton (1998) is based on social learning theory. The 14 items measure perceived level of control (internal or external) over the outcome of consumer events.
The Functional Attitude Scale (FAS)
The Functional Attitude Scale used by GfK was adapted from work by Daugherty, Lee, Gangadharbatla, Kim and Outhavong (2005) and Katz’s early functional theory (1960). Attitudes serve one or more of four distinct personality functions: utilitarian, knowledge, ego-defensive and value-expressive.
The Influentials PS Scale (PS)
The Influentials PS (Personality Strength) Scale was developed by Weimann (1991) and based on Katz and Lazarsfeld’s (1955) two-step flow theory of communication, which asserted that consumers may be more influenced by each other than through media messages. The 10-item battery subsumes three latent dimensions: self-confidence, commitment and leadership.
Our dependent measure was created by multiplying the number of different activities that a person did online or on a wireless device (other than a phone call) by the number of hours spent on online for their own personal reasons, regardless of how or where they accessed the Internet.
Research design
Sample
A deliberate effort was made to select a diverse set of online respondent resources for the study. It was assumed that the diversity would yield the differing distributions of underlying latent class structure.
Source A: Traditional online survey access panel. There is no pay-for-play unless a specific survey length is met or exceeded. Monthly drawings are used to award incentives.
Source B: Survey access panel that mainly uses a referral model to recruit panelists. Incentives, to some extent, are associated with the recruitment referral model as members can earn rewards if referred members join and participate in surveys.
Source C: Members are recruited to earn points for doing a range of activities, including survey participation.
Source D: Social networks: these people joined their respective social networks for their own reasons, not for participating in survey research.
Source E: Portal and other sites that the source has relationships with offer site visitors the opportunity to participate in surveys and earn rewards for doing so.
Approximately 3,600 participants who completed the 35-minute survey were used in this research and analysis.
Results
Latent class regression was used to reveal five underlying classes of consumers. Each unique segment was distinguished by a different Internet usage driver profile. That is, the dependent measure (online engagement) was a function of a different set of personality metrics for each of the five segments. Figure 1 presents the overall model results. Class I represented 35 percent of the sample whereas the smallest segment (Class V) was comprised of just 4 percent of the respondents.
Figure 2 presents the complete model. As shown, many of the PVQ variables emerge as significant predictors of online engagement. Locus of control emerged as a significant negative predictor in Class II, indicating these respondents perceive an internal locus consistent with their self-direction and achievement drivers.
Two predictor variables - benevolence and self-direction - were present in three classes, suggesting they play an instrumental role in online engagement for a large portion of the Internet population. This is particularly true for self-direction, which is a driver for each of the three largest segments.
As expected, the five online communities included in this study differed considerably with respect to their class compositions, as shown in Figure 3. For example, Panel D (social network) had the largest concentrations of Class III and V respondents whereas Panel B (referral model) and Panel C (points-based) reflected the largest proportions of Class I consumers.
The Panel C (points-based) and Panel E (reward-based) class distributions appeared quite parallel - as one would expect. Similarly, the two more traditional online panel communities (A and B) also emerged as very similar.
Critical question
Now, the critical question is: what types of consumers characterize these five latent classes?
From a demographic perspective the five classes differed in interesting ways, as shown in Figure 4. Other than Class V there were no substantive gender skews. Class II consumers appeared to be older, more apt to be married and own their own home. The youngest and oldest groups were clearly Class II and Class V, respectively.
Figure 5 provides a highly-abbreviated snapshot of the past 30-day purchase incidence data. The ranges on these data tend to be relatively wide. Apparel and shoe purchases ranged from a low of 42 percent to a high of 59 percent. In contrast, past 30-day cell phone purchase ranged from 5 percent to 16 percent.
In terms of the types of activities members of the five segments have engaged in over the past 30 days (Figure 6), it’s clear that the very tech-oriented Class V consumers tend to avoid traditional periodical formats (offline magazines) and are more oriented to movies and computer games. Conversely, Class II members appear more traditional, tending to eschew fast food and computer games. Note that Class III consumers closely parallel many of the proclivities of the younger, ostensibly more tech-savvy Class V members.
Figure 7 reveals several noteworthy differences in consumer attitudes. For example, the spendthrift Class V members concede they are on a tight budget. Class IV consumers could be attractive to certain marketers as they appear to be more impulsive and driven by a need for conformity.
A comprehensive analysis of the latent class regression segments is ongoing. The segment profiling phase has revealed many interesting and intuitively-appealing patterns. These brief snapshots confirm that:
Class I members are clearly most affluent and well-educated and their external locus of control may lead to more Web-based information-gathering.
Class II members are older and reported the most online survey activity yet the lowest incidence of purchasing many products.
Class III consumers were also more affluent and notable due to their high levels of benevolence, agreeableness and openness. They also tend to shadow Class V members with respect to certain behaviors.
The most politically conservative group (recall these data were collected right before the fall 2008 election) was Class IV. These consumers scored highest on the conformity and traditional PVQ scales. This was the only group to vote overwhelmingly for McCain.
Finally, Class V members were the youngest and most tech-savvy. They were also the most self-directed with hedonistic tendencies.
Extensions will be explored
We believe this research represents a proof of concept in many ways. Our intent is to administer the full instrument to a large offline sample in order to develop a national model which will subsequently be used to classify 10 or more online communities. Based on this exercise, several extensions will be explored including: panelist survival models and intervention strategies for each class; an a priori classification/recruitment model; unique acquisition and retention strategies for each class; global extension to European markets.
We believe these enhancements to the foundational market structure analysis will yield a comprehensive approach to online sampling that will minimize the impact of each community’s idiosyncratic member acquisition and retention strategies. The benefit clearly involves greater stability and robustness.
References
Bekkers, R. (2004). Giving and Volunteering in the Netherlands. Amsterdam: ICS.
Busseri, M.A., Lefcourt, H.M., Kerton, R.R. (1998). “Locus of Control for Consumer Outcomes: Predicting Consumer Behavior.” Journal of Applied Social Psychology , 28, 12, pp. 1067-87.
Cacioppo, J.T., Petty, R.E., Kao, C.F. (1984). “The Efficient Assessment of Need for Cognition.” Journal of Personality Assessment, 48, pp.306-7.
Carlo, G., Okun, M.A., Knight G.P., de Guzman M.R.T. (2005). “The Interplay of Traits and Motives on Volunteering: Agreeableness, Extraversion and Prosocial Value Motivation.” Personality and Individual Differences. 38 (6), pp. 1293-1305.
Cattell, R.B. (1943). “The Description of Personality: Basic Traits Resolved Into Clusters.” Journal of Abnormal and Social Psychology, 38, 476-506.
Costa, P.T., McCrae, R.R., Dye, D.A. (1991). “Facet Scales for Agreeableness and Conscientiousness: A Revision of the NEO Personality Inventory.” Personality and Individual Differences, 12(9):887-898.
Daugherty, T., Lee, W-N., Kim, K., and Outhavong, S. (2005). “Organizational Virtual Communities: Exploring Motivations Behind Online Panel Participation.” Journal of Computer-Mediated Communication, 10(4)
Glendon, A.I., McKenna, E.F., Clarke, S.G. (2006). Human Safety and Risk Management. Boca Raton: CRC Press.
Goldberg, L.R. (1990). “An Alternative ‘Description of Personality’: The Big-Five Factor Structure.” Journal of Personality and Social Psychology, 59, 1216-1229.
Jang, K.L. and Livesley, W.J. (1996). “Heritability of the Big Five Personality Dimensions and Their Facets: A Twin Study.” Journal of Personality. 64 (3); pp. 577-592
Katz, D. (1960). “The Functional Approach to the Study of Attitudes.” Public Opinion Quarterly, 24 (2), 163-204.
Norman, W.T. (1967). 2,800 personality trait descriptors: Normative operating characteristics for a university population. Department of Psychology, University of Michigan.
Pearson, K. (1894). “Contributions to the Mathematical Theory of Evolution.” Philosophical Transactions of the Royal Society A, 185, 71-110.
Roccas, S., Sagiv, L., Schwartz, S.H., Knafo, A. (2002). “The Big Five Personality Factors and Personal Values.” Personality and Social Psychology Bulletin, 28, 789.
Rotter, J. (1966). “Generalized Expectancies for Internal Versus External Control of Reinforcements.” Psychological Monographs, 80, Whole No. 609.
Schwartz, S.H., Melech, G., Lehmann, A., Burgess, S., and Harris, M. (2001). “Extending the Cross-cultural Validity of the Theory of Basic Human Values with a Different Method of Measurement.” Journal of Cross Cultural Psychology, 32, pp. 519-542.
Weiman, G. (1991). “The Influentials: Back to the Concept of Opinion Leaders?” Public Opinion Quarterly, 55, 267-279.