A return to quality

Editor's note: Doug Berdie is president of Consumer Review Systems. 

I have experienced a sad decline in the quality of marketing research and other social research practices and data since my initial entry into the field in 1974. Back then, researchers went to great pains to obtain representative samples and to ensure that good response rates were obtained from those samples so that data could be confidently generalized to the populations of interest. Care was taken to extensively pre-test the wording on survey questionnaires and to be certain more objective data from records were accurate, timely and could be clearly interpreted. Some of these practices took time and were expensive but because the decisions to be made based on the data had major financial implications, the extra time and expense to get “good data” were deemed well worth it.

As technology moved forward in great leaps from the late 1980s and up to now, methods to obtain data became cheaper and faster and, hence, were readily adopted. Back in the 1970s and earlier, many governmental organizations and businesses would decide which geographic areas to survey by randomly selecting block samples and, then, randomly selecting households within those blocks, using city directories and other sources to do so. They didn’t stop there, though. They selected random samples of people (usually adults) within those households to constitute the sample. And, finally, great effort was made to obtain data from those exact people, which meant if, for example, a researcher doing in-person interviews knocked on the door of a selected home and a different adult answered the door because the selected one was not home, the researcher would not interview the door-opener but would ask when the selected person would be home so a return visit could be arranged. Time and expenses were allocated to follow-up techniques (in this case, return visits to the home) to ensure high response rates would be obtained and, hence, reliable data. In-person interviews and mail surveys were used most commonly because, at that time, representative samples could not be obtained by telephone interview given the disparities of telephone ownership.

Once telephones became more universal among homes, focus shifted to telephone interviewing. Researchers realized that bias still existed to some extent due to unlisted numbers and careful researchers conducted small reliability checks to ensure this bias was not too large. Sadly, the focus on obtaining survey data from a representative group of people started to shift to the easier task of getting it from samples of households – not worrying so much about who in the household provided the information. This assumed that all household members held the same attitudes and had the same background experiences upon which those attitudes were based – which was clearly not the case.

The most recent shift based on technology arose with widespread internet access. Using e-mail and websites it became easier to reach large numbers of people quickly and inexpensively. Sending out thousands of survey invitations was a fast and cheap way to get a few hundred completed surveys. And, research panels allowed firms to start data collection at any time to address emerging issues. 

Each of the shifts in data-gathering associated with changes in technology has led to a degradation in data quality – the price for speed, reduced cost and ease. And, these less-than-desirable practices have gone on for so long that they’re now accepted and, as a result, some major decisions have been based on suboptimal data. 

Following is an examination of why that is and what best practices exist to help us return to high-quality data.

Wrong and misleading

Most marketing research reports and political polls make a point of stating how precise the data are. These “results are accurate within x%” statements are almost always wrong and misleading – for many reasons. Theories of probability that underlie statements of data precision require 1) an initial random sample; 2) a high response rate from that sample; 3) data from clear and unambiguous sources; 4) calculations based on the exact type of question asked or other source of data; and 5) reporting results clearly and correctly. Let’s look at each of these requirements.

Random samples

As noted above, clean random samples are rare these days – and yet they are required to calculate real precision estimates. Obtaining a true random sample involves first identifying all the members of the population of interest and then using random techniques to select a sample. Identifying the members of the population of interest is easy in some cases (e.g., the people who purchased a new vehicle from a dealer during a given time frame) and less easy in other cases (e.g., people who went shopping in a major metropolitan downtown on a given day). Years ago, while doing a survey for a city, we needed to station workers with clickers at randomly selected corners of a downtown to estimate how many people actually did enter the downtown at those locations during a specified time period and then projected those numbers to the entire downtown. By acknowledging that this count was an estimate, we could present the subsequent data as just that.

Response rates

Sending out 10,000 survey invitations over the internet and having 600 people respond and complete a survey does not result in data from a random sample. If that practice is used, at a minimum one should find some characteristics that may be known for the entire population (e.g., age, income, years as a customer, etc.) that can be asked in the survey. That way, comparisons can be made between the total population and the survey respondents. However, one must keep in mind that representativeness in terms of these demographic characteristics does not ensure representativeness of response in terms of attitudes, recent experiences, etc. In customer satisfaction research, those who are most dissatisfied and those most satisfied are often overrepresented in the survey respondents – even though their demographic characteristics may be similar to the nonrespondents who have more ambivalent attitudes. 

The best practice to ensure responses are representative in all regards is to obtain a high response rate. This used to be much more common than it is now. A literature search I conducted found 74 research studies where response rates from 80%-100% were obtained during the 1924-1988 time period and I am aware of many additional ones during that time frame and since then that have reached those levels. Methodological research I‘ve published found that mail survey response rates above 50% and telephone survey response rates in the 65%-75% range generally provide representative results. But it is important to reach, at a minimum, those levels. Those results are based on having compared the resulting data at those response rate levels to “what the data would have been had there been a higher response rate.” 

The second measure resulted from using extensive response rate-stimulating follow-up techniques to boost the response rate much higher than the first measure. This is a best practice for assessing possible response rate bias and is much better than comparing data from early respondents (as a group) to initial nonrespondents who have been followed-up with and successfully gotten to participate. After all, the real question is, “Would we make different decisions had there been a higher response rate?” rather than, “Do people who initially did not respond differ from those who responded at the outset?”

When designing research projects, it is far superior to select small random samples (that are large enough to meet desired precision levels) and to expend time and resources to use follow-up techniques to attain higher response rates than it is to use very large samples and “just take what comes in” without follow-up. The response rate bias that results from the latter techniques is usually large enough to negate what had been hoped for in terms of sample size precision.

Also, the all-too-common practice of using fill-in samples does not protect against response rate bias. This practice “replaces” non-respondents with additional sample which, even if randomly selected, does not lead to the same results that are attained with good follow-up practices to get the input from those in the original sample who do not respond initially.

Data sources

For all marketing (and other social) research that asks people to respond to questions, it is essential that the questions asked: be clearly and unambiguously understood; be understood in the same way by all who answer them; be worded fairly – i.e., not be biased; and avoid a variety of other question-wording flaws. An example of a poor question I once asked was, “Do you believe there are enough cultural opportunities in this neighborhood?” People did answer the question but we discovered later they had not all interpreted it in the same way. When the data showed a large percentage of people had answered no, we had follow-up discussion and asked what they’d like to see more of. Some people said they’d like more varied artistic events and others said they’d like more restaurants serving foreign food. We realized the word “cultural” had been interpreted in varying ways and we had no idea what percentage of people had done so in one way versus another. Better pre-testing of the question before administering the survey would have caught that problem so it could have been fixed. Stanley Payne’s classic book, “The Art of Asking Questions,” provides a great list of most flaws in question wording and should be required reading for all marketing researchers. (The book I co-authored, “Questionnaires: Design and Use,” also has tips on problems to avoid.) 

The raw input underlying marketing research obtained in ways other than surveys (e.g., historical records) is susceptible to a variety of interpretations and must be examined carefully with that in mind.

Analysis tied to types of data and levels of measurement 

I cringe when I see precision estimates presented as one number – e.g., “The data are accurate within +/- 4%.” For such statements to be true all questions in a survey must be of the exact same type – which is rarely the case. For example, a question with only two response options requires a different formula to calculate precision than does a question with five options. A question asking for a number (e.g., “age in years”) requires yet a different formula to calculate precision. And, as if it weren’t complicated enough, the formula used to calculate data precision for a given type of question (e.g., a yes-or-no question) will yield different precision estimates if the percentage of yes responses is different for the one question than the other – even if the number of responses is the same. So, in reality, precision estimates need to be calculated for each question in a survey and should be presented right by the results of those questions. A best practice is to present the results for each question or variable in ranges (e.g., “Between 42%-48% of people answered Question #1 with a yes.”). And, yes, to make it easier on decision makers who must deal with the results, one can offer summary statements (rather than drowning them in data) such as the following: “In almost all cases, the percentages shown below are within five to nine percentage points. But the responses to Question #6 are much less precise than that, so please examine those data carefully.”

Reporting data precision properly

Unfortunately, many research reports incorrectly report the precision estimates for categorical questions as a “+/-%” when, in reality, it should be “+/- percentage points.” If one says the precision for a yes-no question with 38% “yes” responses is “+/-5%” one is really saying the range indicating the estimated response is “36.1%-39.9%” (38 x .05 = 1.9, which is subtracted from the 38% and added to the 38% to get the range). The formulas used for this type of question, however, generate estimates of percentage points, so if the formula generates a “+/-5” it is really saying the estimated range of response is, “38% +/-5 percentage points” (or, “33%-42%”) – quite a difference from the incorrectly stated version. This is one of the most common flaws in reporting data.

As noted above, different types of questions yield different precision estimates. So, asking a question that elicits a number (e.g., “age”) yields a +/- number that is in the form of the unit of measure. If the average age is found to be 37 it will be reported as something like, “37 years +/- 4.2 years.”

Bonus: A good political tip 

When random samples are selected for surveying, there are times when those who were not selected get upset: “Hey, I heard you asked John Jones for his opinion. Why didn’t you ask me? My opinion should count too!” If you are in the type of research situation where you might encounter this, consider using the “sample + everyone” approach. It consists of selecting a scientific random sample, earmarking that sample so it can be identified, surveying everyone and tracking the returns from the earmarked random sample so nonrespondents in that group can be followed-up with. Then, when a high response rate is obtained from the earmarked sample, those data can be analyzed separately and used as the basis for decisions. It’s also a good idea to compare those data to the data obtained from all the people surveyed who responded. The two data sets are almost always very similar and in cases where they are, you can present the data from the larger group, knowing that the random-sample data validate data from the larger group. This way, “everyone” had an opportunity to voice their opinions and potential political anger can be avoided.

Easy to lose sight

Lots of effort has been expended over hundreds of years to define and refine the theory of probability and use of language so that accurate marketing research data can be collected, reported and used to drive solid business decisions. As businesses evolve, and ways that data can be collected become more sophisticated, it is easy to lose sight of the fundamentals that underlie credible, accurate marketing research data. And yet, no amount of technological glitz can offset these fundamentals. Those entering the profession need to be exposed to the best practices described above so we can ensure marketing research remains on a solid footing.