Selecting a consumer panel service
Editor's note: Brian Wansink is professor of marketing, nutritional Science, and agricultural and consumer economics at the University of Illinois at Urbana-Champaign. The late Seymour Sudman was the Stellner Distinguished Chair of Research at the University of Illinois at Urbana-Champaign. This article is based insights gathered during the writing of Consumer Panels (by Seymour Sudman and Brian Wansink; published by American Marketing Association: Chicago).
Given the widespread use of consumer panel techniques in marketing work, nearly every marketing executive or researcher concerned with consumer goods will have to consider at one time or another how to use this technique in their work. Choosing a consumer panel service then becomes of considerable importance, since it is far cheaper to buy into an ongoing consumer panel than to set up a separate panel for one's own use. Although the latter procedure guarantees confidentiality and exclusive control, it also entails large expenditures, as well as the necessity of recruiting a special staff - which very few manufacturers or retailers care to incur.
It therefore seems appropriate to outline the principal factors to keep in mind in selecting a consumer panel service. Moreover, considering such questions as how to determine the most suitable consumer panel for a particular purpose and how to evaluate the relative merits of panels also helps to clarify what is really needed from a panel study.
Eleven tips in choosing a panel service
Although the principal points to consider in selecting a consumer panel service vary to some extent with the type of product and with the problems to be solved, certain basic factors are applicable to all types of panels, including continuous purchase panels, product testing panels, copytesting panels, and others. These factors are brought out in this article in the form of a series of 11 pointers that should be followed in considering a consumer panel service. It should be noted that all of these pointers relate to the objectives and the technical aspects of the panels, not to the cost. This is because trying to save some out-of-pocket costs and thus running the risk of getting highly unreliable information is a case of being penny-wise and pound-foolish. In using consumer panel information - regardless of whether collected online or offline - the objective should be either to obtain highly reliable and appropriate information for the problem or not to spend any money at all.
1. How could these results possibly change what I would do?
Our training as researchers and as businesspeople has always told us that more data is good. While it is no surprise that data comes at a cost, data is most valuable when it informs a decision. Suppose we are trying to find out how people shop. If the range of possible outcomes of this research will not alter any decision we will make, the research should not be done. Unfortunately, much non-diagnostic research is conducted every day. It usually begins with someone saying, "Wouldn't it be interesting to know..." The problem is that when the resulting crosstabs, bar chart, or pie chart is presented, part of that year's research budget is gone and managers are not any wiser than they were prior to the research.
One way to avoid this is to proactively specify a wide, discrete set of findings or outcomes that could occur if one did the research. For each of these outcomes, write down what the resulting managerial action would be for each outcome. If the managerial actions (or responses) are the same for each of the outcomes, there is no need to do the research. If there is a wide variation of what your managerial response would be, this exercise will probably also suggest additional questions you will want to ask.
2. Choose either static or dynamic panels
Having a panel composed of the same consumer units or individuals throughout a given period makes a static panel especially useful for evaluating the effects of advertising or other stimuli. By providing before-and-after purchase records for the identical panel members, a static panel helps to keep extraneous factors constant and brings out more clearly the effects of any experimental treatments used.
If, however, panel data are wanted to serve as a basis for making generalizations about the market, a dynamic sample is necessary. Such a panel will provide for rotation of the consumer units so that the distribution of its members is kept in line with current shifts in population characteristics. Such a rotation pattern should be on a staggered basis and should allow for estimates to be made for the population based on the composite results of both the old and the new segments of the panel.
In choosing between these two types of panels, it is useful to keep in mind that it is always possible to incorporate a static panel within a dynamic panel, but the reverse is not possible.
3. Don't confuse panel data with one-time survey data
When it comes to one-shot surveys, online versions save time, eliminate interviewer bias, offer better international coordination, provide multi-media capabilities, and can help improve data quality. Pop-up definition boxes and relational navigation links make the survey more understandable for respondents and easier to complete, and versing, skips, rotations, and piping allow for more complex questionnaire design, and enable conjoint and pricing studies to be completed more efficiently.
While these advantages are attractive for one-time surveys, they may not be worth the costs for continuous panels. For certain panels, time saving is not necessarily an issue because they are reported on a regular interval, and the results are seldom urgent, but more long-term in nature. For panels, many of these features may not necessarily be worth the trade-offs that might have to be made relative to sample representativeness or data accuracy. Representativeness was covered above; let us address data accuracy.
The primary objective in panel research is to find consumers who will consistently and accurately complete the entries in their panel diary. Hard-copy diaries are always available (perhaps sitting on a desk) when entries need to be made. If online use is irregular, or if computer and Internet access in the household is a competitive resource, a more convenient form (such as hard-copy panel booklets) might provide more accurate and consistent responses.
For continuous panels, panel members become efficient because the format of the instrument is static and the directions are familiar. The features that make the Internet effective for some types of surveys are less beneficial - and may even detract from a panel's effectiveness. When assessing the necessity of features, it is important to make certain that they are features that will be worth the costs or trade-offs associated with them.
4. Double-check panel representativeness
Be sure that the panel is representative of the particular population to be studied and with regard to the population characteristics that need to be analyzed. To say that a sample is representative of a population without specifying in what respect it is representative is meaningless. What you invariably want is a panel that will yield accurate purchase estimates for each of the consumer characteristics that may affect the sale of the product. Representativeness for one characteristic does not ensure a panel's representativeness for other characteristics. It is therefore important to make sure that the panel is representative with respect to all of the characteristics to be studied.
It is especially important to make sure that the panel is concentrated in the areas among the population groups that are the principal purchasers of the product. Thus, a broadly representative national consumer panel may be wasted for a manufacturer of products that are sold primarily in rural areas. A concentrated panel is likely to be cheaper per dollar expended, and it also permits a more intensive analysis of purchaser characteristics.
It is important to check the representativeness of the reporting sample rather than that of the mailing sample. Consider mail diaries: Since response rates vary by population groups, a perfectly representative stratified distribution of families on the mailing list will not yield a perfectly representative distribution of reporting families. A competently run consumer panel should contain a disproportionately high number of lower-income, poorly-educated families with a low response rate. Remember that the purchase data received will be based on the reporting panel, not on the original mailing list.
Request each panel operator bidding for your patronage to provide the number and type of sample controls used to keep the sample representative and the date of the population estimates on which these controls are based. A progressive panel service will try to keep the sample representative in a great many different ways (if it is a stratified sample), and it will be continually revising its controls to keep them up to date with the changing characteristics of the population. The latter is especially important when large population shifts are taking place. Indeed, this is the only way of maintaining the representativeness of a dynamic panel.
5. Double-check data reliability
Inquire into what the panel service is doing to determine the reasons for not reporting and to assure the accuracy of the purchase reports of its members.
Both of these points are important potential sources of bias, and they require continual checks. No panel is perfect in these respects, and none ever will be. But a progressive organization will do continuous studies and checkups on these points and should be able to show concrete evidence of this work.
A progressive panel operation will also be conducting research and making tests on methods of increasing the accuracy of the purchase reports. Low accuracy of reports is perhaps the major problem of continuous purchase panels, and it is a problem that characterizes even families who are initially fully cooperative. Hence, continual checks need to be made on the accuracy of the reports and on what biases are occurring as a result. If possible, have the panel services submit per-family or per-capita sample purchase figures of products that are related to your own and whose sales can be checked independently. Without divulging these figures, ask your researchers to prepare comparable purchase estimates from the production and/or sales records of these products. Then compare the two sets of estimates. The more reliable your estimates, and the closer these related products resemble your own, the more accurate this procedure is in indicating the relative accuracy of the competing panel services in your case. Needless to say, a similar comparison may be made if any of the panel services have past records on the consumer purchases of your own product.
Keep in mind also that if the primary interest is in measuring the flow of a product from the warehouse to the consumer, a store or warehouse inventory panel may be preferable to a consumer purchase panel. For food and drug products, for example, a statistically valid panel of food and drug stores is likely to yield a more accurate picture of purchase flows than a consumer panel. However, if the primary interest is in correlating changes in purchases with the characteristics of the purchasers of the product, a consumer purchase panel is clearly indicated.
6. Pay only for the precision you need
Precision can come in two forms. One is multiple questions asked in multiple ways that triangulate on the same construct (i.e., using a Likert scale, a semantic differential scale, and an estimated purchase frequency question to measure purchase intentions). The second form of precision is obtained by getting a larger and larger sample, thereby reducing sampling error. The entire concept of sampling is based upon the notion that it is infeasible and unnecessary to survey an entire population. So too is there a point at which a sample size is large enough to effectively answer the questions for which it is intended.
While precision is good, after some point there is less and less value to being more and more precise. The cost of asking multiple questions for every construct of interest can be measured in a lower response rate and higher error due to fatigue. On the back end, this cost can be measured in increased data handling and increased analysis and reporting.
To ask something because it "might be interesting" is expensive. The more diagnostic the questions, the better and the greater the need for precision. One rule of thumb is to ask how much the outcome in a measure would have to change before it would change a decision. If a change from 4.0 to 3.9 would result in a product being launched, or an old product being dropped, there is a high need for precision. That is, a large sample size and multiple questions are necessary. If, on the other hand, the change from an estimate of 4.0 to 2.5 would still not change a decision, the sample need not be as large and multiple questions might be trimmed.
7. If accuracy is critical, measure sample error
Determine whether you want sample data that will permit you to measure the sampling error. In most cases, estimates of the sampling errors in the purchase data are extremely desirable. Without such estimates there is no way of knowing to what extent sampling variations may have introduced errors in the purchase figures. Unfortunately, such error estimates are not possible with some continuing consumer panels because their members are not selected in the true random fashion that is the basic prerequisite for the applicability of sampling error formulas.
A knowledge of the sampling error is not so important for some purposes as for others. Thus, in general a knowledge of the sampling errors is not so necessary in studying trends as in "blowing up" the sample data to population estimates. However, even in the former case, much greater reliability can be placed in a panel whose sampling error is measurable.
Further, if a panel is selected whose sampling error can be measured, be sure that the panel will yield results within the required sampling error limits for strata and substrata as well as for the total sample. As a rule, a major requirement is to have a sample that yields acceptable error limits at the smallest levels of aggregation. The specification of strata and substrata, as well as acceptable sampling error limits for these entities, are frequently the critical considerations for sample size. It makes a huge difference, for example, whether a sampling error of five percentage points at the 95 percent confidence level is desired separately for each of four different income levels and three different age groups or whether the same sampling error is desired for the combinations of income level by age. In the latter instance, minimum adequate sample sizes have to be considered for each of 12 income-by-age strata, which would require a sample many times larger than that needed for minimum sampling errors for income and age strata separately.
This also means that one should not ask for a lower sampling error than is absolutely needed, since generally the lower the sampling error, the higher the sample size and the higher the cost of the service.
8. Think sample reliability, not sample size
Don't rely on sample size alone as a determinant of the reliability of the results. Other important considerations are the sampling method used to set up the panel, how the individual panel members are selected, and how the data are collected. The sampling method is particularly important. A small randomly-selected sample is likely to be far more reliable than a much larger arbitrarily-selected sample.
The principal determinant of the reliability of the sample data is not the sampling variance but the mean square error. As a rule the bias component of the mean square error tends to be many times that of the sampling variance in consumer purchase studies, so that primary attention has to be given to the nature and magnitude of the nonsampling errors inherent in the panel operation. Therefore, it is important for a panel operation to make constant checks on the accuracy of the purchase reports and on other sources of nonsampling error.
9. Stratum sample size is more important than general sample sizes
Be sure that the panel is large enough to supply reliable purchase figures for each of the breakdowns required. Not only must the panel service be able and willing to supply the necessary breakdowns, but there must also be a sufficiently large number of members in each stratum and substratum for which figures are desired. Exactly what constitutes a sufficient number depends primarily on the popularity of the product: the less widely purchased it is, the larger the number of sample members required in a particular substratum to yield a cross-section of the characteristics of its purchasers.
Generally, it would be wise to insist on at least 25 panel members in each stratum for which purchase data are requested, although at times this figure may be much too low. For example, if only 3 percent of the families in a certain city purchase Product X, a panel substratum of 25 to 30 members in that city may easily contain not a single purchaser of the product.
10. Sorting out the offline vs. online decision
For some situations and some questions, offline panels are the obvious choice. For other types of questions, online panels are obvious. The difficulty lies in the gray areas. Each method has its benefits and each has its self-selection biases. The crucial concern with an online panel is that it is presently the least favorable choice for research in terms of accuracy, according to Karl Irons, president of NPD Online Research.
Online panels can have some tremendous benefits when the population is accurately sampled. Online panels are fast, they can generate large samples, they can be inexpensive (no mailing and printing costs and lower labor costs), they can show graphics and video, and they can provide seamless international coordination. Furthermore, they can reduce in-house errors associated with interviewer bias or coding and data entry errors.
Despite these benefits, there are concerns that Internet-savvy consumers are not representative of the general population. While no method offline or online is perfectly representative of the population being studied, there is a sizable concern that online panels are psychographically biased toward progressive technology innovators, and demographically biased toward young male professionals. If this is true, then the typical grocery shopper is probably represented more accurately by offline panels then online panels. In this case, the decision of which type of panel to use to study grocery shopping would involve a trade-off: the speed and savings of online or the increased accuracy of an offline panel. For certain questions, accuracy is less important than others.
Yet the representativeness of online panels to the general population of consumers is only important if you are actually interested in the general population of consumers. There are situations where the population of interest - for instance, people who purchase on the Internet - is best captured by an online panel. Presently, items that can easily be bought using the Internet (books, compact discs, airline tickets, magazine subscriptions, home banking, investment services, and software, etc.) are probably good candidates for online consumer panels.
Furthermore, there are subgroups within the population that may be better represented through Internet than through other methods. Teenagers, for instance, have been an elusive group with respect to consumer panels prior to online consumer panels. This is also true for consumers who are single and for well-educated audiences, such as doctors, lawyers, and other professionals, and it may also be true for working mothers.
In summary, the online vs. offline decision depends on the products and the populations you are studying and on the level of accuracy that you need. As Internet use increases, biases will decrease, and online panels will be better able to access a more general population.
11. Don't ask for more data than you need
The more data you request, the more you pay - not only for the panel data but also for the time your researchers spend in analyzing and cross-analyzing the figures. In the long run, it pays to concentrate on those consumer characteristics that affect the sales of your product most strongly. To accomplish this, try to obtain purchase reports for your product by as many different characteristics as possible the first few times. By analyzing these reports, you can determine which characteristics have the least influence on the purchases of the product and so could be dropped from future reports - except, perhaps, for occasional checks, if desired. The savings resulting from this procedure could be spent on obtaining more intensive breakdowns of relevant characteristics. For example, if the purchases of Product X vary greatly by region and by city size but not by income or occupation, the two latter classifications might be dropped, and a larger number of region and city size breakdowns could be requested, either a finer breakdown of regions and city sizes or a breakdown of purchases by city size within regions.
It is especially desirable to keep the number of printouts requested to a minimum, since they can multiply astronomically when comparisons are being made not only by population characteristics but also over time. Unless the office is in danger of being blown away by a tornado, it is wise initially to ask for printouts on a very selective basis, since the sheer volume that can be turned out by a computer in the batch mode is enough to discourage even the hardiest of researchers from plowing through them. For exploratory purposes a highly efficient (although not the cheapest) means of selecting relevant characteristics and key tabulations is to have the panel data online and to produce alternative frequency counts and tabulations interactively using a computer. Many forms of multivariate and more sophisticated analyses can also be carried out in this manner, and numerous alternatives can thereby be sifted out without having the office take on the semblance of a paper warehouse.
Summary
It should be clear from the foregoing that the best panel service is not necessarily the cheapest one. To save a few dollars at the expense of reliability may jeopardize the value of the entire operation. Most important of all is obtaining the greatest reliability for your dollars. If you subscribe to the panel service that can give this to you, any additional expenditure for that service will be well worth the cost.