Do market research in the market

Editor’s note: Herb Sorensen is president of Sorensen Associates Inc, a Troutdale, Ore., research firm.

For many years, the twin guards of integrity at the doors of market research have been security and past participation screening of respondents. Both are geared to eliminate the “wrong” kind of respondent from research studies: security to eliminate competitive leaks; and past participation to eliminate “professional” respondents -- those who are too familiar with the research process. On reflection, these two screening criteria are closely allied.

Regardless of the merits of this or that approach to security and past participation, long-standing suspicions and anecdotal reports of failures of these twin guards have recently been confirmed by solid research. Careful studies demonstrate that less than 5 percent of the population produces more than half of all the responses used in market research (Bickart and Schmittlein, Journal of Marketing Research, May 1999).

One response to plummeting cooperation rates by consumers has been to try to educate and persuade the public that their self-interest is involved in market research; as well as to identify consumer-friendly research practices (CMOR, see www.cmor.org/cmorrespach.htm). Another is to conclude that past participation doesn’t matter and to accept “professional” respondents as the norm. This latter position has been given powerful impetus by the on-line survey community, who tout huge respondent panels but are in fact repeatedly surveying what must be a very small slice of their panels. A minimum of monthly surveys are recommended by these practitioners, and some of their respondents are actually being queried multiple times per week!

However, it is not necessary to acquiesce to the slumber of the sleeping guards. Both security and past participation criteria can best be attained through a sensible statistical approach. Some caveats and observations are appropriate:

1. Absolutely secure consumer research is very nearly an oxymoron. “You mean you want me to find out what a thousand people think about this new idea, but you don’t want anyone to know about it?”

2. Standard security questions on intercept screeners are probably nearly worthless. The very people you most want to eliminate from a study are the ones most likely to misrepresent themselves in order to get access to the study. Reporting on a recent MRA/QRCA discussion and study in the July 1999 issue of Quirk’s, Joseph Rydholm noted that: “While the exact scope of cheating/repeating is tough to quantify, anecdotal evidence suggests the problem is common.”

3. Several years ago a major packaged goods company was conducting a “highly secure” study on the eastern seaboard. Research facilities were carefully selected and audited for security to assure the tightest possible restrictions. All evidence suggested that these precautions were effective, until the firm’s own sales manager in one region submitted a detailed report on the new concept. No one in the sales chain (including this manager) was privy even to the existence of the research. This manager’s wife, by chance, had been asked to participate in the study and provided the details for the report, making it appear the firm had stumbled onto some major competitive initiative.

4. Approximately 10 percent of the staff of our firm has access to studies fielded through the largest mail panel company in the country. This is through mothers-in-law, wives, etc. A client who works with us regularly felt that they needed the “added” security of a mail panel for a sensitive new product concept. We, of course, were unaware of the study’s existence until we received a full set of the survey materials through one of these back-door sources. It’s hard to tell whether the client was glad to learn of this lapse, or preferred to bury it and continue to pretend that they were operating with very tight security.

5. Anytime you interview at a site that is frequently used for market research, by yourself or others, you can be sure you are getting a lot of “professional” respondents. The field services are glad to get these people. They cooperate to fill any quota and are skilled at cheating on security and past participation questions. Reuse of the same site over and over by the research community is guaranteed to produce this problem. One facility in the New York metro area (operated by one of the largest field service chains) is so large that one might conclude that they are an anchor for the mall. (The same principle operates but is controllable in an ethically operated mail panel. By the way, who is independently checking these panels?)

This is just a brief overview of potential and actual security and past participation problems.

The statistical approach to security recognizes that we are unlikely to reduce potential lapses to zero. The method depends on decreasing the odds that undesired parties will accidentally get involved in a research study; and insuring that we will have access to as wide as possible a sampling of our target market.

Begin by thinking about where you plan to conduct your next survey. OK, how many other researchers, possibly competitors, are planning to go to the same place (site or list) for their studies? Rule #1, you shouldn’t be there.

This rule has been virtually ignored by the research industry for decades. Rather, access to respondents has been organized to facilitate the production of large quantities of data in a milieu convenient to the researchers. For the most part, this means data collection occurs in an office environment: the mall “office,” the telephone center “office” or the post “office.”

Why not do your market research in the market where you sell your goods? There you would likely avoid competitive security interests, as well as the pesky “professional” respondent. This would require working evenings and weekends in retail stores to meet the shoppers and avoid tradespeople. Moreover, do not work in the same store more than once in a three-month period. This would typically solve both the security and past participation requirements of the study in a natural way.

Have you ever personally been interviewed in a store? Or have you been “malled,” as one client put it? The first event almost never happens, the second is common. Thus in-store interviewing is the essence of statistical security, and it also solves the problem of past participation. If you collect your data in the market, you are statistically unlikely to have problems with either of the twin guards.

If you are serious about market research, get out of the office and into the marketplace.