Take these steps to build your research on a solid foundation

By Mark Hardy | November 1, 2010

Reading time: 8 minutes

Abstract

Beyond getting the right sample, researchers conducting online surveys face a host of obstacles, from harder-to-reach consumers to the speeders and cheaters who lurk among the pool of willing respondents. Here are 10 topics to cover with your sample provider to help improve your odds.

Research Topics:: Data Analysis | Data Processing | Low Incidence Research | Panels-Diary | Panels-Mail | Panels-Online | Panels-Telephone | Quantitative Research | Respondent Cooperation/Satisfaction | Sampling | Telephone Interviewing/CATI | The Business of Research | Validation-Respondent
Industry/Market Focus:: Research Industry
Content Type: Research Methodology

Share Print

10 questions to ask before choosing a sample provider

Editor’s note: Mark Hardy is chief strategy officer/managing director, Americas, Survey Sampling International, Shelton, Conn.

Solid market research is the foundation of every business’s most critical decisions - and the quality of that research depends on the quality of the sample. A representative, valid, unbiased sample is essential for research results to provide accurate reflections of the market - and reliable guidance for business direction. A host of factors, however, from the explosion of the Internet to the fragmentation of media usage to the rise of social networks, has made it increasingly difficult to attract the right participants and create the optimal sample.

Even traditional RDD (random-digit dialing) landline phone sample - long considered the most methodologically-sound for survey research - is coming into question with the rise of mobile phone usage. The latest National Health Interview Survey shows that 51 percent of U.S. homes are now cell-only or cell-primary households - making them hard or even impossible to reach through traditional RDD. The growth of wireless is not just a U.S. phenomenon. Research by our firm shows that around the world - from the U.K. to Spain to Japan - more people now own cell phones than landlines, and that gap is largest among 18-24-year-olds.

The challenges can be just as great, if not greater, in the online world. The traditional paradigm for online research has been sending e-mail invitations to potential participants. But that paradigm is no longer enough to sustain research into the future. Since 2003, e-mail use for personal messages has plummeted 41 percent, according to the Online Publishers Association. Although 90 trillion e-mails were sent over the last year, 81 percent were spam. It is harder than ever to get participants’ attention when their in-boxes are overflowing with junk mail.

But the decline in e-mail is just one of many changes transforming how researchers need to reach and communicate with participants. Participants now can take surveys anywhere, on a plethora of devices and often while doing other tasks. In fact, our research shows that consumers worldwide now media multitask regularly, often texting, chatting on the phone and surfing the Web all at the same time.

In this new world, technology has given birth to a wide range of new sampling options and sources - as well as new threats to data quality and integrity. To be sure market researchers get sample they can count on to drive the right business decisions, they need to ask questions - and, more importantly, demand answers - different than in the past. Understanding the responses to the following 10 questions can ensure that researchers make the right sample choice for their projects.

1. How are reach and diversity achieved?

The reality is that only a finite number of people will ever join a panel. Although panels always will remain a critical part of the access mix, it is increasingly difficult to deliver all the participants needed for a survey from panels alone - particularly when looking for hard-to-reach targets. In fact, in today’s world, no one source can deliver the reach and diversity critical to unbiased sample.

The optimal sample taps into a variety of sources - traditional panels as well as social media, online communities, affiliate partners, reward programs, shopping portals and more - to provide true reach and diversity. In our multimedia, multitasking world, it is important to engage people wherever they are - and to include all participants, even those who would never be part of a managed panel.

2. How are multiple sources blended?

Blending sample from multiple sources lets researchers reach all people who want to share their opinions - even those not on panels - maximizing diversity, the most important characteristic of a representative sample. Blending also creates a better sample by improving coverage and ensuring that the opportunity to take surveys is placed in front of as large and varied a population as possible. Blending, therefore, actually can result in a better-quality sample than using any one source alone - that is, when it’s done right.

When choosing a sample provider, confirm there are quality controls in place, such as digital fingerprinting to avoid duplication. Plus, verify that blended sample is regularly checked to reflect changes in source composition and market dynamics. It is also important to ensure the provider considers a full range of factors by retesting in a multisource environment to ensure balanced sample, adding calibration questions to surveys to help explain differences and employing smoothing techniques.

3. What is the recruitment approach?

In today’s sampling world, variety and flexibility are key. Diverse sourcing - critical to unbiased sample - demands eclectic recruiting. Effective sample providers use a variety of recruitment methods to drive traffic to surveys, rather than, for example, bombarding people with pop-up ads. By matching recruitment methods to partner sources and their membership, sample providers can both increase participation and improve the participant experience.

4. How are participants treated?

People are vastly different in what motivates and engages them. Even more importantly, when they feel they are being treated fairly and their efforts are appreciated, they provide better data. That’s why your sample provider should nurture participants. Effective sample providers treat their participants like their clients. They make expectations clear, answer questions quickly and offer reward systems as varied as their sources, with options ranging from point systems to sweepstakes to charitable donations to information to sincere thanks. Fully customizing rewards ensures sample providers successfully motivate each target audience.

5. How is the participant experience managed?

In a world where there is constant competition for participants’ attention, it is more critical than ever that we create survey experiences that are positive and engaging. One of the biggest obstacles to participant satisfaction - and the largest sources of participant fatigue and frustration - is being screened-out from surveys.

Screen-outs happen when participants are ready and willing to share their opinions - but just at the moment they want to participate, they are told they don’t qualify. Screen-outs happen when sample providers screen participants for one survey at a time. Many people seeking to participate will not meet the criteria for an individual project - and will find themselves shut out of the process.

To avoid that negative experience, it’s important to seek sample providers who offer participants many projects for which they could qualify, screening for multiple studies at once and thus greatly increasing the chances that people who want to complete a survey will have that opportunity. This approach reduces screen-outs, increases participant satisfaction, slashes the number of e-mail invitations and cuts drop-out rates.

6. What processes are in place for identifying speeders and cheaters?

Technology has created new ways for people who want to game the system. Fortunately, it also has enabled the creation of powerful tools to protect against this type of fraudulent activity. Research shows a very small number of participants intentionally try to cheat on surveys. Nevertheless, it is critical that sample providers have proven techniques in place to prevent any type of fraud that would compromise data integrity. Among the tools and processes you should ensure sample providers are implementing to protect quality include:

Timestamps to flag participants who have completed a survey - or a portion of a survey - too quickly to have provided relevant responses.
Checks to identify straightliners - participants whose answers remain static across a survey (all As, for example) or the same pattern of response (such as ABCABC, etc.).
Quality-control questions to catch participants who are not paying attention, are inconsistent in their demographic information or are not following instructions.
Matches against third-party consumer databases to confirm each panelist’s name, address and date of birth, making sure all participants are who they say they are.
Database analyses to identify and remove fraudsters in real time. People trying to complete surveys fraudulently - using false identities and providing answers just to collect rewards - exhibit common behavior patterns, such as very fast survey completion times and out-of-area IP addresses. They also tend to use their varied identities in a consistent pattern across surveys, as well as to qualify for surveys with very different target audiences. As a result, it is critical sample providers have tools in place that identify these behavior patterns so can eliminate fraudulent responders.

7. How are participants validated and de-duped?

In today’s world, where survey participants come from multiple sources, it is possible for people to be invited more than once to the same survey. Therefore, often unintentionally, a person may try to respond twice to the same questionnaire. For quality data, it is essential that sample providers have controls in place to protect against duplicate participants.

Digital fingerprinting is one tool that is critical for preventing duplication. Digital fingerprinting identifies each participant’s machine. This is done through watermarking (a sophisticated type of cookie that cannot be easily removed) or through tracking multiple data points (such as system time, screen resolution and software versions). When a person logs on to take a survey, the machine’s ID is screened against all those already on file. If a duplicate is found, the participant is not able to take the survey.

In addition sample providers should use a variety of techniques to authenticate participants. These can include traps to identify geo-IP violations, address matching (such as, in the U.S., matches against the USPS postal file) and profile-specific queries that only the legitimate participant would know how to answer.

8. How are Web partners chosen?

With multisource sampling, providers integrate information from many Web partners to create a balanced sample. When choosing a sampling vendor, make sure it shares its process for ensuring each partner provides quality sample. Reliable sample providers have a consistent set of standards they apply to evaluate sources before incorporating them. They should fully vet each source to confirm it provides a positive participant experience and contributes to providing a fully representative sample.

9. What modes of access are available?

With all of today’s communication options, the people needed to complete a research project can be tougher than ever to reach. They may be online or offline, wired or wireless, Internet-savvy or Web-averse. Therefore, depending on only one mode to fill sample may mean missing out on a critical segment of the universe.

Our multimedia world demands multimode sampling - particularly for lower-incidence targets. Ask if sample providers under consideration can offer access through a range of online and offline modes, as well as through mixed-access approaches. This is particularly critical for projects with small universes, narrow parameters or hard-to-reach audiences.

10. How is science applied to ensure representative, balanced samples?

Sampling is not just about filling quotas. If the sample is not balanced, unbiased and representative, the information it delivers can be inaccurate - and misleading. Make sure vendors can provide methodologically-sound sample plans before beginning a job. Plans should include solid selection techniques; detailed stratification and targeting; precise geographic and demographic allocations; rewards that motivate; appropriate contact methods; and active panel and community management programs.

Take the time to ask

A solid sample is a critical foundation for effective research. With so many forces converging to transform how people seek and share information, it is important to evaluate sample providers against new standards - ensuring they can deliver quality sample in our new world. Take the time to ask sample providers the right questions - and demand complete answers. Ensuring quality sample is essential to ensuring effective research results that guide accurate business decisions.