Methods for ensuring online sample quality around the world

Know your markets

Editor’s note: Chuck Miller is president of research firm DMS Insights, a uSamp company, in the Dallas office, and chief research officer of uSamp, an Encino, Calif., research firm. He Suresh Subbiah is chief operating officer, uSamp, in the firm’s Trumbull, Conn., office.

With the continued expansion and maturation of online market research, quality of online panels and rivers continues as an area of focus for the market research industry. Naturally, the industry has high expectations for its evolution. A great many initiatives are underway to advance quality, aggregating the collective intelligence of industry thought-leaders. All of these are valuable and critical for producing better-quality data. Beyond current efforts around respondent validation and de-duplication (among others) some key questions remain:

• To what degree do these efforts produce quality data, especially on a global scale?
• How does sample differ in North America, Europe and APAC and how does this impact quality?
• Can the industry standardize panel (and river) quality globally?

Quality in the research world is a complex equation, with a number of factors in play. To address quality concerns adequately, researchers and panel operators must keenly understand myriad factors to produce high-quality global research.

To produce quality online sample globally, expertise is required in four key areas: understanding differences in capabilities and cultures on a global scale; standardizing global sampling processes, while allowing for regional differences; leveraging technology appropriately to improve standardization; and engaging global survey takers through a variety of methods.

1. Understanding differences in capabilities and cultures on a global scale

Given vast differences around the globe, grasping regional differences is no small feat. The combination of differing Internet characteristics (penetration, usage, etc.) and cultural characteristics (such as attitudes and behaviors) makes solving for best practices quite complex. Obviously, one size does not fit all.

As such, sound evaluation and a nuanced approach are needed. The bar may be higher here for panel operators, even more so than for those executing global surveys. Without fundamentally sound sample, even the best study execution will yield inaccurate results. As such, sample providers must thoroughly understand every market in which they work. There is no substitute for regional knowledge, as it allows the sample provider to determine practicality and best approaches to sampling and interviewing in a region. The next step is applying this knowledge to produce the best sample given each situation and consult clients on regional capabilities and limitations - including when to supplement online interviewing with offline approaches.
Reinforcing the value of regional knowledge, consider these examples:

• Online studies requiring a census-balanced gen-pop sample are now commonplace in the U.S. As such, many researchers may unreasonably assume this can be done in countries with large Internet populations like India and China. But while the number of Internet users is very high in these countries, the composition still skews toward urban users in larger cities.
• In some areas, such as the United Arab Emirates, the floating population is high - meaning the profile of locals vs. expatriates varies greatly. In these situations, it’s essential to understand the target audience in light of the research objectives (i.e., targeting current residents or indigenous populations). These countries require more frequent updating of panel profiles, given the more transient nature of the population.
• In countries like Japan and France panelists are very sensitive to how they are treated and survey quality and communication are critical. Regional knowledge around social norms, scale usage and response interpretation is critical.
• Beyond general sensitivities, some topics simply may not work on a global scale. In particular, research related to some medical conditions can be problematic. Additionally, care should be taken to understand legal implications of certain topics - some may not be possible to inquire about in certain countries.

Most of us realize that sampling and survey approaches that work in the U.S. may not work in other markets. Simply translating U.S. English into other languages is not sufficient, providing yet another reason why regional specialty is critical. Demographics, economy, online penetration and culture all influence respondent characteristics and subsequent data quality. Among these influences, culture most frequently tends to be overlooked.

2. Standardizing global sampling processes, while allowing for regional differences

As noted, regional expertise is critical for creating and maintaining a high-quality sample. Global panel management needs and processes for consumer, B2B and specialty panels are different and require differing strategies by country. Because certain aspects of panel management can be global while others must be local, the key to success is establishing the right blend of practices and then applying standardization both locally and globally.

In particular, online sample providers should look to standardize certain elements in every engagement: panel and/or river recruitment processes, with an emphasis on sourcing stability; respondent registration processes, including consistency of data collection and user experience where appropriate and possible; respondent profiling, including questionnaire elements and communications; sample selection, including invitation and usage rules.

Most important (and often overlooked) is the means of standardizing sample recruitment and sample frame construction. Research data is only as good as a drawn sample, which in turn is only as good as its sampling frame. So, fundamentally, to produce good research it’s necessary to produce a solid and consistent frame - as well as employing solid sampling practices. Consistent sourcing of sample is critical and, while it may seem obvious, standardization pays dividends here.

The industry is at a critical point in its evolution: in many countries, panel operators are facing declining response rates (generally due to panelist overuse and/or long or poor surveys). In other countries, Internet research is still new and novel, and as such response rates remain solid. In many of the mature markets, the combination of increased demand for panel and potentially challenged supply leads some to grow and refresh panels through any means available.

An effective strategy is for panel operators to employ strong metric-driven business rules and standardized processes to forecast then grow capacity, in a way that maintains consistency of their sampling frame. Without this attention to the fundamentals, the ability to replicate sample is questionable - even if consistent business practices and rules for sample usage are employed. When there’s too much flux in panel sourcing, drawn samples and research results will fluctuate as well.

Interestingly, while the industry recognizes this in its discussions about river sampling, it doesn’t hold panels up to this same level of scrutiny. Our hope is that as river sampling continues to gain momentum (a technique, we might add, that has been done successfully for nearly 15 years now), the industry will apply this same level of consideration to sourcing of all international panel and river creation. When that occurs, everyone will be better off; standardization produces consistency of sample sourcing and frame construction - elevating both the science of sampling and the quality of output.

3. Leveraging technology appropriately to improve standardization

Once practices and guidelines around standardization have been established, technology can be leveraged to ensure enforceability and consistency. Standardized applications and methods deployed around sample recruitment and management have a big impact on quality. Companies that successfully design and deploy applications not only create process efficiencies, they also improve quality along the way.

That said, recalling that we must always apply a nuanced and local vs. global approach to sample management, it’s important not to apply technologies and standardization carelessly, just for the sake of achieving efficiencies and consistency. For example, while common in developed nations, tracking IP addresses to improve quality can be problematic and potentially biasing in many regions. In countries where Internet connectivity is not yet pervasive, many people share connections in places such as Internet cafés and libraries. As such, employing a hard, one-size-fits-all block on IP address is unwise.

Conversely, there are a number of areas where certain technologies, consistently applied, will help improve quality. Among these: recruitment monitoring and throttling by source, to maintain a consistent sampling frame (for both river and panel); registration processing, to ensure respondent quality and validity; respondent profiling, to ensure consistency, validity and completeness; invitation response and survey monitoring, including tests for inherent quality.

Across the board, sound business rules and technology practices will help create scalable global sampling and research. Expertise in applying technology optimally - both globally and locally - is critical. For instance, in the IP-tracking example, decision-making and expectations for chief privacy officers are clearly elevated for global panel operators. In such areas, regional differences must be taken into account when establishing practices; in many European countries, IP tracking has legal implications and restrictions. It is important to fully understand and appreciate these differences within the scope of technology deployment and in any quality initiative globally.

4. Engaging global survey takers through a variety of methods

Even when successfully identifying global and local needs, then applying standards and technology to ensure best practices, an organization’s efforts can be meaningless if it doesn’t appropriately engage its audience in an ongoing manner. A number of considerations here can affect the research itself, as well as the long-term health and viability of the online samples.

First and foremost, it’s vital that not all research be done online. Especially in lesser-developed nations, there’s still a need for mixed-mode data collection (including face-to-face) when attempting to reach a broad cross-section of the population. If online capabilities aren’t a fit for the research objectives, force-fitting the method does a huge disservice to both respondents and clients. When online sampling alone is not sufficient, seek offline supplements (or even replacements). In those situations, care should be taken to: complement the online research, by adding sample subsets not readily available online; understand potential sampling and methods differences, and their implications for the results (e.g., conducting a portion online and conducting a portion with an interviewer); and consult with clients on options (and implications) to arrive at the best solution.

Beyond this primary consideration, other factors weigh on maximum engagement:

Communications with respondents. These obviously vary due to language, but cultural norms must be considered as well. This is where local or regional expertise is most relevant. Word-for-word translations are generally problematic, creating unknown and unintended consequences, which often affect results.

Survey designs. To be engaging, survey topics, length and on-screen presentations require review on a country-by-country basis. Global studies should be carefully designed, factoring in language (including regional dialects), regional sensitivities, cultural implications and characteristics of available respondents. Likewise, prevailing practices around desktop (or mobile) technology must be considered - this is more important globally than in the U.S. Surveys may need to be constructed for lowest-common denominator technology usage among participants.

Motivation for participation. Respondents participate in research for a host of reasons: curiosity/interest, altruism and rewards, to cite just a few. Whatever the incentive, the objective is to complete the highest percentage of a drawn sample, while providing nominal gratuities that attract without biasing. As such, rewards for participation should be consistent (but not necessarily consistent on a monetary basis) to obtain the best data. If significant differences exist in panels or methods, datasets with vastly different characteristics may result, thereby hindering cross-country comparisons.

Maintaining a high level of engagement is both art and science, requiring high degrees of knowledge and experience. This effort also requires that sample providers, research agencies and end clients work in concert to make the right choices for any particular study, as well as for the sample sources and their participants, in the long run. While somewhat daunting, successful engagement is key to delivering high-value research now and in the future.

Aligning multiple players

Building and nurturing high-quality global sample sources involves diligence, thoughtfulness and an aggregation of regional experiences. It’s a process of orchestration, of aligning multiple players in concert, including interaction with high-quality surveys that produce solid research results. This complex task is ongoing, as Internet adoption continues to evolve around the globe. Here, collective intelligence leads us to the best outcomes. As such, it is essential that, as an industry, we be continually vigilant to expand expertise and knowledge. In doing so, we best serve clients, protect global research assets and deliver accurate and actionable research.