From evolution to a resolution

Editor’s note: Dennis Q. Murphy is vice president, technology practice at Directions Research Inc., Cincinnati.

Do I believe that there is a respondent cooperation issue? Let me reprise an anecdote included in my keynote address at the 2002 CASRO conference.

As the then-worldwide vice president of market intelligence at IBM, I frequently reviewed work done around the company. In mid-2000, a major division brought forth a project that they’d conducted, frustrated that insightful findings were few and faint. Thirty minutes into the conversation, I broached methodology and response rates. Methodology employed was “floppy disks via mail.” Further, the return rate was two out of three - three hundred that is! Seven-tenths of one percent! My initial reaction was to castigate but cynicism won out. I congratulated them. “Nice work guys. You’ve successfully isolated the lunatic fringe.”

Cynicism is not how I normally interact with the world but this topic does bring it out. Response rates have been declining my entire professional career yet, until the announcement of the September 2006 Respondent Cooperation Conference (see Trade Talk, p.138, Quirk’s, November 2006), lip service is about the best we’ve done. If we’re serious about resolution then we first need to embrace the entire problem. The following are some of the elements that have contributed.

Evolution

Environmental change

  • Migrating societal mores in an increasingly complex world have made respondents suspicious of anything that looks like an invasion of their privacy. In the early 1970s we could go door-to-door and be served milk and cookies; today we’d be served with trespassing papers.
  • Telephone became the default methodology of choice. This medium diminished interaction, eliminated visual and tactile opportunities, and shortened the window of respondent attention by (my estimate) at least 50 percent. In all fairness, telephone methods are just one point on the line from in-person to Web surveys to scanner data. The less in-person the method, the less personal and the less vibrant the data.
  • Water’s no longer free and coffee’s not a nickel. Respondents recognize today that their opinion “has value” and are reluctant to participate without remuneration. Time is money and we’ve trained them to think that their opinion is worth something.

Market pressures

  • Some clients, blessed with sufficient resources and insightful management, have maintained traditional research standards but more have not. Tight client budgets and procurement involvement in RFPs have led to increased low-cost bidding. In turn, many suppliers have felt coerced into “bare-bones proposals.” This has taken a toll on potential respondent quality with diminished rigor in both sample design and recruitment. And clients, for the most part, have watched passively.

One “corner-cut casualty” of cost management has been rigorous sample design. The precise block-by-block sampling plans of door-to-door have not been easily replicated with other methodologies. With cooperation already down, relying even more heavily upon the few who are willing to cooperate exacerbates the problem. It is hardly surprising that even these few diehard responders are now on the wane in hard-to-recruit markets like IT professionals, physicians, etc., as our oversampling alienates them as well.

  • The complementary concern is how well these few to whom we speak speak for the many we don’t. Re-sampling to estimate non-response bias was once de rigueur; today it’s got rigor mortis - and with tight budgets and less-than-rigorous sampling it’s not a likely candidate for resurrection. So with declining respondent cooperation, the fundamentals of projectability - representative sample and non-response estimation - are MIA. Said otherwise, today all samples are convenience samples and projectability is a crapshoot.
  • Fewer business executives recognize or demand rigorous research. It becomes more expedient to simply execute “a few groups” or “get some down-and-dirty information” than to wait and make good decisions with strongly designed work. These soft sources are non-threatening while a sophisticated market model can be confrontational to the technophobic decision maker or to managers who simply don’t subscribe to the “science of management.”

Self-inflicted wounds

  • Supplier researchers are loath to share any limitations or inadequacies of their work with client researchers lest they turn to an alternative supplier. Client researchers are similarly loath to share “research problems” with their internal customers lest they axe the work altogether. So what we have is market research’s dirty little secret: the respondents might not be of the quality promised and their responses unarticulated as well. While the ramifications of open conversations are frightening, those of continued closure are terminal for the profession.
  • Migration from in-person interviews to dinner-time phone calls, promised at five minutes and delivered at 20, have decimated research credibility. That these calls may not be market researchers - perhaps telemarketers or other solicitations - is a subtlety of little relevance and less interest to the public.
  • We design surveys with shopping-list linearity to acquire what the clients want to hear with little regard to facilitating what respondents have to say. Guiltlessly, we then execute surveys that demand more than respondents can reasonably deliver.
  • The resulting survey experience itself is often time- and mind-brutalizing. Forced to take our own surveys - which we should be required to do - we’d often blanch. All respondents should expect an engaging survey experience and the video game generation will demand it.

I’ve used the word “dilemma” in this article’s subhead but I’m fine with “crisis” and even better with “calamity,” for that somehow captures that we’re the author of this drama. That said, I don’t believe the situation to be funereal. We simply have to begin dealing with “what is” rather than “what we would still like it to be.”

Resolution

Our first requirement is to agree that it is broken and it does need to be fixed. If we do that, then a plan can be constructed. The following points are at least a place to start.

Agreements

1. Raise the bar on quality. Our intention should always be to deliver high-quality information which is germane to those who will receive it. This demands that we design screeners that can discriminate between marginally and maximally qualified respondents. Even when doing qualitative, we need to promise that we’re truly speaking to the right folks.

2. Recognize that respondent incentives are the new norm. It’s time to admit that we’re a gratuity-based business and figure out what that means and how best to assess the impact. In most of the world, the days of “free information” are history.

3. Revise standards and guidelines. Quantitative market research has been co-opted by other information resources - sometimes appropriately and sometimes not - but regardless it needs to reassert its unique ability to create a robust market view. Qualitative is a noble partner in exploring and defining relationships but only quantitative brings measurement and explanation. Our survey work depends upon rigor - in sampling, design and analysis - and such discipline would enhance the qualitative world as well. Some of the actions below will elaborate.

4. Honor the profession. Complementing new standards must be some method of “good behavior” recognition individually and collectively. While industry-promotional efforts aimed at the respondent public must be considered, such programs tend to be pretty impotent at the expenditure levels within our means. That doesn’t mean however that seals of approval, Web sites and other recognition devices aren’t useful - just incomplete. The real mark will be made by how we interact with respondents in the future, not by what we have to say.

Actions

1. Defend the role of quantitative research. We’ve failed to position qualitative work as largely exploratory in contrast to the projectable and prescriptive potential of quantitative. We must reestablish quantitative market research to its full potential - the process that brings dimension to market information and explains relationships amongst variables. Academia needs to help as well. They need to subjugate the mechanics of research (statistics, models, etc.) to the core contribution we make - management of research and coaching decision makers. The former may be what we do but what matters is what we affect.

2. Deal with each market as it exists today. Big CPG markets haven’t been oversampled beyond repair but learn from those that have. Many B2B markets are so saturated that either no one is left to talk to or some underemployed guy named Joe with nothing better to do than respond has become the de facto industry spokesperson. If a market no longer can yield anything representative, just be upfront about it and look for a new avenue.

3. Make sampling the art of the possible. Put forth more effort into designing screening devices that can meet representativeness testing. When that fails, consider panels or other approaches. Though a long-time panel critic, I obviously believe it’s better to build panels of available respondents and filter out learning bias as best as possible than to have no one to talk to at all.

4. Recall non-response bias. Maybe we can’t reinstate the old 40-percent rule but consider that when response rates dip below 10 percent, just maybe some non-response work is called for. Perhaps that becomes a mandatory guideline for inclusion in proposals - with client option regarding necessity. At least it would now be aboveboard.

5. Treat the phone as an excellent medium for very short encounters. Telephone work is great for political polling and other fairly binary, non-visual work. It also can serve well for recruitment to other mediums. After some fairly brief interlude - certainly beyond 15 minutes - the value it delivers deteriorates rapidly. Respondents become random-number generators. We’d be better off if they just hung up.

6. Employ the real value of the Internet - creative execution. We’ve adopted the Internet well for recruiting - which it does poorly - while underemphasizing its executional power, which can be truly transformational. The Internet’s design capability is “CATI to the nth power” yet much Internet-executed research simply analogs some of the most boring and mundane paper-and-pencil approaches ever devised. We need to recognize the potential of this resource and get beyond bottom-feeding applications.

7. Reconsider face-to-face work. While this may be largely cost-prohibitive today, perhaps we can incorporate more rigorous survey procedures into what are otherwise qualitative occasions.

8. Enhancing the respondent experience will enhance respondent cooperation. Let’s target interestingly designed surveys as the price of entry and then reposition surveys as an entertainment medium as the first milestone. Interesting and enjoyable surveys are both more fun to design and more fun to do. Let’s elevate respondents from resource to hero. Let’s also produce goal-driven research where we supplant “shopping-list work” with a network of questions encasing a desired outcome or decision point. Finally, let’s aspire to devising whole new paradigms. How about designing games that are research in disguise?

9. Be cautious when attempting to educate the public. Entertained, amused, tempted, rewarded - okay. Seals of approval and other such programs which are low-cost are fine but we can’t rely significantly upon an outbound program. “Educating the public” is too often the battle cry of those about to go down in flames. This is our problem, not theirs. Hence our mantra might be: Ask not what respondents can do for you but rather what you can do for respondents!

Stem the tide

Individually, these agreements and actions will stem the tide; collectively they should reverse it - our dirty little secret bleached clean. Wouldn’t it be great to stand in front of a room sublimely confident that the respondents were perfect and the information impeccable? Wouldn’t it be nice to not have your fingers crossed behind your back with every contention you bring forth?

Three years ago my wife and I moved from the New York metro area to a lake in rural Minnesota presuming I would retire - a pursuit at which I’ve obviously failed. While personally not an outdoorsman, I’ve developed a begrudging tolerance for those who are. One interesting facet of the hunting community is that while they prey, they also protect. Certainly one could argue that their wetlands protection program, Ducks Unlimited, is self-serving, but then aren’t most efforts that actually work?

The research community could do far worse than to take a page from such organizations. It’s time for market researchers to launch Respondents Unlimited.