Editor’s note: Ellen Pieper is vice president at market research firm Symmetric, Dallas.

I recently saw a presentation by Consumer Reports that discussed its large consumer survey. In the days before the Internet, social media and online reviews, people would look at Consumer Reports’ reviews before making a big purchase. Consumer Reports gathers reviews through a consumer survey answered by 800,000 to 1 million consumers every year. Consumer Reports’ survey is second only to the U.S. Census, and that is done only once a decade. The presenter shared two interesting facts about Consumer Reports:

  • First, whenever the speaker tells someone that he works for Consumer Reports, the first reaction is, “Oh, my grandparents read that.”
  • Second, Consumer Reports once used both a mail methodology and a Web methodology but they have eliminated the Web methodology. 

Given these two facts, I wondered how they could possibly achieve a representative sample. Based on the presentation, the average age of the mail survey respondent is 80 years old, which is in line with the people in their target audience: your grandparents. So they don’t need to worry about having a representative sample per se. 

Sample balancing for the rest of us

The Consumer Reports situation is far from typical. What about the rest of us? If we can’t rely on 80+ years of history to know our target audience, or if we’re looking to attract a different audience, how do we ensure a representative sample? This is where balancing comes into play. Balancing a sample is simply the process of drawing your sample to ensure that it reflects the population of interest. Let’s look at three types of sample balancing.

  1. Balanced sends: This methodology harkens back to earlier days when mail and phone were the primary sources of data collection. It is no longer considered a good methodology for ensuring representativeness – unless you are looking for those over 70. The logic behind this method of balancing assumes that people of all ages, genders and ethnicities are equally likely to respond. Now we know that different groups respond at different rates, so this method has fallen out of favor.
  2. Balanced starts/clicks: This is the balancing methodology to use when a researcher is trying to define their target audience. In balanced starts/clicks, you can see who is qualifying for your study and that enables you to define the target audience. 
  3. Balanced completes: The most common type of balancing starts with quotas on the completes that mimic the Census (or balanced to your known target audience). Researchers oversample certain groups to achieve these quotas, as not all groups respond in the same amount of time (for example, young men are slow to respond, if they respond at all). This methodology is also known as quotas on the completes.

When deciding what kind of balancing is right for your study, you need to consider the research objectives.  What are you trying to learn? Answering the questions below will help you decide what kind of balancing makes the most sense for your study: 

  • Do you already know the market or user demographics?
  • Is the study trying to identify who the product users are? 
  • Do the sample demographics need to be aligned with the target market?
  • Is the study trying to compare groups of product users to another product’s users (or non-users)?
  • Is the study a tracker that depends on trending data?

We often use a combination of balancing methodologies. If you start your study using balanced completes you are going to be forcing sample in a somewhat unnatural way (unless the incidence will be consistent across demographics, which is almost never the case). Take Consumer Reports, where only “grandparents” read the magazine. If Consumer Reports has a quota for younger people, they will have to screen through many more people to find those who qualify (i.e., read the magazine), thus lowering the overall incidence rate of the study. However, if you do want ensure a good spread, you can set some quotas that are proportional to market share, which is determined by using balanced starts/clicks. 

Application 

While good in theory, actually appliying these methods gets tricky. Let’s go through two examples:

  1. Balanced starts – completes fall as they may (natural fallout): In this project your raw sample statistics reflect the market or population statistics and you filter and screen respondents without making or forcing adjustments. Through the project you learn the demographics of the key market simply because of qualification. You can still look at longitudinal trends but with two caveats. First, there is a higher risk that shifts in the sample supply may cause changes in the trend. Second, a chance variation in the trend may be more influential than with other types of sample balancing.
  2. Balanced completes – starts are adjusted to get the completes needed: In this type of project, raw (unweighted) sample stats don’t estimate the population parameters. Rather, the population parameters are known in advance and the sample is drawn to meet certain quotas. Balanced completes sampling limits what we learn from the screening and qualification process. We don’t learn the demographics of the target audience because we predefine it. We don’t pick up the shifting demographics of the market. However, it also has some benefits. There is a lower chance that there will be variation in the distribution of the completes. If this is a tracking study, waves will have much higher longitudinal stability. Further, it’s the best way to match previous results. It also gives researchers more latitude to switch sample suppliers between study waves.

Tips and recommendations

If you know the market structure and you are worried about stability, have quotas on completes (balanced completes). If you want to reveal market structure based on screening, use balanced starts. If you want to both learn the market structure and have stability, here are three steps to follow: 

  1. Program quotas on completes but don’t set strict limits or quotas. 
  2. Send balanced starts, complete between 25 and 33 percent of your total responses and then look at how your data is falling. 
  3. Adjust quotas based on the distribution of your completed surveys and complete the rest of your study. If the distribution of demographics was acceptable, no change is needed. If the structure is unacceptable (e.g., too few in some groups, too many in others), adjust the quotas to achieve a more even distribution. You also can set quotas to ensure the results are nationally representative.

It sounds a little difficult to do, and indeed it is. Perhaps samples should come with a warning, “Developed by trained professionals. Do not try this at home!” However, if you work to understand the sample plan that best meets your study’s objectives you should be able to avoid the pitfalls that are inherent in poor data validity and reliability.