There is a law, and it may help you choose the best method for the job
Editor's note: Fred Phillips is professor and department head, Department of Management in Science and Technology, at Oregon Health & Science University, Beaverton, Ore.
Word got out that you have a market research problem. Two vendors have appeared at your door; one sells surveys, and the other sells panels. Which should you choose? If you choose a panel, what kind of panel will cost-effectively solve your problem?
If you want to measure a trend - in purchase volume, market share, or any other behavior - you'll want to give panels a serious look. Even then, deciding among surveys and various flavors of panels is no slam dunk.
This article expands on Wansink and Sudman's excellent essay in the May, 2002 issue of Quirk's1, giving trend watchers an advanced primer on panels as well as more tools (and one simple "law") for making this important decision. The tools extend those in Sudman's classic Consumer Panels2, which he wrote with the support of MRCA Information Services (then called Market Research Corporation of America) while consulting for the company. Later, when I was MRCA's vice president for research, the present article's tools were devised and published in academic journals.3
Suppose you need to know the month-to-month trend in a company's purchases of software licenses, or a household's purchases of breakfast cereal. The January-February trend in purchase volume is simply the difference between the amount purchased in February and the amount purchased in January. Trend measurement necessarily involves a span of time - a before-and-after - and is an example of longitudinal measurement. We can contrast this kind of measurement with cross-sectional measurement, which involves only one point in time. (Examples of cross-sectional marketing queries include, "What brand did you buy today?" or "Whom did you vote for in the last election?" and so on.)
The word "panel" refers to a research design in which respondents answer the same questions at least twice, with the reporting occasions separated by an interval of time. An ideal panel is one without attrition, i.e., one in which all respondents in the first wave of reporting also report in subsequent waves. The population of respondents in a panel with attrition can be static or dynamic. In a static panel there is no replacement of panel members who have dropped out after the first wave, whereas a dynamic panel is replenished in order to preserve the panel as a good representation of a particular population. The total replenished panel sample, including both original respondents and replacement respondents, is called the summary sample. As an alternative to panels, a trend might be measured by two or more independent, cross-sectional surveys, each with a different sample of respondents.
Operators of continuous, multi-client panels replenish their panels as a matter of course; they may then charge extra for the programming needed to "pull" a static sample. It is widely presumed that static samples are superior to summary samples for minimizing sampling error. We show below that this is not always true, and that research users (who often pay a premium to ensure a sizeable static sample) can save money by using appropriate summary samples.
To briefly review Stats 101, both surveys and panels are samples, subject to sampling errors of two kinds. Sample bias arises when we sample a group of respondents that is not really representative of the target population of buyers. Sample variation happens when we sample representative respondents but not enough of them.
The word "bias" means the same thing in statistics as in ordinary language; a biased sample statistic is "a little bit off" its true value. Momentarily ignoring sample variation, we can write the measured value of the trend like this:
Measured January-February trend in purchase volume = (True volume in February + bias in February measurement) - (True volume in January + bias in January measurement)
Panels have a clear advantage over repeated independent surveys, insofar as bias is concerned. In a panel that is ideal or nearly so, January bias pretty much equals February bias, and the bias terms in the above equation cancel out, leaving only the true trend. When two surveys use different sampling methodologies and/or measure completely different sets of respondents, the two bias terms differ, and the survey-measured trend is in error by an amount equal to "February bias minus January bias."
Should we measure a trend using repeated independent samples, or a panel sample? Because bias is hard to estimate (most commercial researchers provide a speculative list of possible sources of bias, without estimating their magnitude, and call it a day), we will make decision rules based on the variation of the estimate, or more precisely, the mean square error. A key element of our decision is the serial correlation (sometimes called autocorrelation) of respondents' behavior. The serial correlation reflects the tendency of individual buyers/respondents to repeat the same behavior in successive measurement periods.
If a market measure is reported in two independent surveys of size n taken in two periods, and this market measure has the same variance Ó2 in both periods, then the mean square error of an observed trend in this measure is 2Ó2/n. When observations are taken on an ideal panel (i.e., no attrition) for two sufficiently short periods, and the serial correlation of individual behavior is p, then according to Sudman, the mean square error of the measured trend is:
MSE (ideal panel) = 2Ó2 (1-p)/n
This formula implies a well-known proposition:
For any positive p, the MSE is less for an ideal panel than for repeated surveys.
That is, the ideal panel is preferable. Now consider panel attrition and its effect on measuring the trend using only the static (non-dropout) portion of a panel with attrition.
MSE (static portion) = 2Ó2 (1-p)/Rn
where R = proportion of non-dropouts (the "retention rate"), and n = size of the summary sample. R, of course, is the complement of the attrition rate A. (R = 1 - A.)
This implies a second proposition, which may be called a "law" because of the simple and unexpected nature of the result:
The static panel estimate is less variable than the survey estimate whenever p > A, i.e., when the autocorrelation exceeds the attrition rate.
Because such simple relationships are rare in marketing (at least, useful ones are), my colleagues have kindly dubbed this "Phillips' Law of Longitudinal Sampling."
There are a number of simple proofs of Phillips' Law. It can be shown further that MSE (summary sample) = 2Ó2(1-Rp)/n. Simple arithmetic then shows that:
The estimates from the summary sample are less variable than those obtained from the static sample whenever p < 1/(1+R).
This inequality holds for broad classes of consumer goods, leading to a conclusion that violates conventional wisdom: Summary samples are often preferable to static samples.
How can a market research department put these tools to use? The key is the autocorrelation statistic, and sadly, the exact value of this is not known. However, research users should ask vendors to estimate the autocorrelation of buying behavior in the market of interest. Rough estimates can be gained by looking at other studies done on the same or similar product category and the same or similar target market segment. (Salespeople for research vendors won't be accustomed to being asked this question! You will probably have to ask to talk to the vendor's research director.) Plug the estimated autocorrelation into the inequalities presented in this article; you will then be able to decide among independent surveys, static panels, and summary panels - and get the most accurate trend estimates for your budget.
References
1 B. Wansink and S. Sudman, "Selecting a Consumer Panel Service." Quirk's Marketing Research Review. May, 2002, pp. 30-36.
2 S. Sudman and R. Ferber, Consumer Panels. American Marketing Association, Chicago, 1979.
3 Y. Wind and D.B. Learner, "On the Measurement of Purchase Data: Surveys vs. Purchase Diaries." Journal of Marketing Research, 16 (February), 39-47;
B. Golany, F. Phillips, and J.J. Rousseau, "Optimal Design of Syndicated Panels." European Journal of Operations Research 87 (1995) 148-165;
B. Golany, F. Phillips and J.J. Rousseau, "Few-Wave vs. Continuous Consumer Panels: Some Issues of Attrition, Variance and Bias." International Journal of Research in Marketing, 8, 273-280, 1991.