Editor’s note: Jason Thomas is president of market research firm Symmetric, Dallas.  

Multi colored people figuresLike so many aspects of marketing research, the sampling landscape has changed dramatically in recent years. Many client-side researchers now use DIY platforms to conduct marketing research that is simpler to manage – for example, when using customer or general population samples. Researchers often turn to supplier partners for the rest of their research needs. This often leaves more challenging studies (long-term trackers, advanced analytics) and hard to find or difficult to reach samples (e.g., B2B, geo-specific, low incidence) for the full-service marketing research firms and sample providers. 

Sampling itself is becoming more difficult for sample providers. Despite new sampling techniques and new ways to reach people, the pool of people willing to join marketing research panels or wanting to take surveys while providing high-quality responses is dwindling. And let’s not forget the ever-increasing pressure to complete marketing research projects faster while maintaining data quality. It’s a real challenge.

However, if researchers understand how the sampling industry works, they can make the best possible sample choices for each project, matching the project to the most appropriate sample and/or sample provider.

How the sampling industry works

When you purchase sample from a provider, there are three common scenarios you might experience:

  1. A sample providers has its own sample database or panel, known as proprietary sample. It is built, recruited, managed and owned by that one sample provider. These sample providers may be able to fill your sample quotas using only their proprietary source.
  2. A sample provider partners with other companies to fill project quotas when a proprietary sample does not have enough qualified respondents, combining multiple sample sources into one.
  3. A sampling company does not have a sample database or panel. Instead, the company completes all projects by reselling or aggregating other providers’ samples and by leveraging their knowledge of many different sample sources and offering unique technology. 

The first item of business is to understand which scenario you are facing. You should ask your sample provider if they have a proprietary sample. If so, sample providers need to be completely transparent about their sample characteristics. And sample providers need to be equally transparent if they are partnering with other providers on the research project.

Sample blending

Let’s complicate things a bit more. Respondent pools are recruited using different methods and sources: online, offline, in person, telephone, mail, etc. How the sample is recruited introduces a bias into the sample. Therefore, all sources have biases.  

When filling a challenging or hard to reach sampling quota, sample providers often partner with other sample providers to fill the project. This is known as sample blending. Sample blending is neither good nor bad; it is simply an artifact of our times. Sample blending has advantages and disadvantages at the marketing research project level, as shown in the chart below. 

Pros of Sample Blending
Cons of Sample Blending
  • Allows sample providers to fill difficult samples that they might not be able to fulfill using their proprietary sample alone.
  • It allows the survey to be completed in a reasonable or possibly shorter time frame than by using proprietary sample alone.
  • Sample blending helps erase biases you might have in a single sample source.
  • Diversity in sample sources can protect against poor sample performance and poor quality. If you have one partner not performing well (low quality, low response or completion rates, poor targeting abilities), you have a “built in” backup.
  • Sample blending gives you a low-risk way to test new partner sources by allocating a small part of the project to vet their performance.
  • If a marketing research company or end client is using multiple sample providers, and has not specified they want only want proprietary sample, sample providers could be using similar partners to help fill the study. This could cause duplicates and delay the project.
  • Duplication in feasibility may also cause the full-service marketing research firm to overcommit (multiple sampling partners may be partnering with the same firms, resulting in over counting).
  • Abruptly changing your sample source can impact your sample’s representation or trends, which is especially dangerous from wave to wave in tracking studies.

Sample bias

Sample providers can perform well on some targets and not on others. By using different partners, you can minimize the impact of biases in any single sample source. Further, working proactively with your sample providers to blend your sample can result in significant benefits for your project, such as increased efficiency, shorter field times and lower costs.

Each proprietary sample has an inherent bias or skew as a result of the recruiting mechanism. When online research took off in the late 1990s, many panels were created with the sole purpose of people signing up to voice their opinions and take online surveys to help improve products (while earning something of value). Times have changed over the past 20+ years, and most panels are formed with some other focus in mind, with taking surveys being a secondary focus. 

For example, some providers build panels with people who sign up to earn more playing time for online games, so these panels might skew younger or mobile-heavy. Some panels were built using travel reward programs, so respondents may skew toward higher incomes. There is nothing wrong with these approaches. But you must understand the pros and cons of the sample and how it might affect your results. Blending sources – or allowing your sample provider to blend them for you – will help erase the bias of using one single sample source. If you can leverage the biases inherent in any sample source to assist in meeting your project’s goals, you can make sample blending work for you.

Communication and transparency

Communication is relatively transparent between sample companies but seems to be lacking when looking at similar communication between clients/marketing research firms and sample providers. Sample providers may need to educate their clients about the origins of their proprietary sample and specifically which providers they are bringing on to assist them. Sample providers need to proactively initiate the discussion with full-service marketing research companies and client-side researchers about whether they are fully feasible with proprietary sample only or if they bring on sample partners to help fill the study. And, of course, sample providers must respond honestly when asked questions about their own sample or about blended samples.

Full-service marketing research companies and client-side researchers must open a dialogue with sample providers early in the research process, preferably while specifying and bidding the project. It is the responsibility of the researchers (both full-service marketing research companies and client-side researchers) to be informed sample buyers and complete their due diligence. Consider these questions:

  • Do you need proprietary sources only, or can your sample provider use sample partners? Which option is best for your project? 
  • What partners are most appropriate? If more than one sample provider is used, do they overlap (e.g., using the same third-party resources)? 
  • Will you allow river sampling and, if so, what percentage of your sample should be from river sampling?
  • How will these various sample sources impact fulfillment for your project?
  • How will these various sample sources impact data quality for your project?

Remember, once you start data collection, it is too late to learn that your sample choices were not appropriate for the project. Over time, you will get to know your sample providers, partners and the strengths and weaknesses of different samples so you can use them to your advantage. As you learn about new sample sources, plan to blend them into your known sample sources so you can safely become more familiar with them without compromising your results. 

While a lack of communication about sample blending could cause unintended consequences, blending is a positive sample approach overall. Sample blending allows difficult samples to be filled successfully. Blending allows researchers to complete research projects more quickly. But transparency and clear communication are necessary among the key players in a project to increase efficiency and avoid unintended consequences. Transparency about sourcing will lead to greater understanding, as long as everyone takes responsibility to improve the situation.