As they argue the health and viability of MR, those pushing new approaches and those resisting them need to take a long look at their positions. Both are commonly using flawed logic.
Editor's note: Ron Sellers is president of Grey Matter Research and Consulting, Phoenix.
The debate over the future of the consumer insights industry continues in full swing. On one side, we have the Disrupters – people who project that surveys will be gone in a few short years, that next-generation approaches such as big data analysis, AI and biometrics will entirely replace primary research, etc. On the other side, we have the Traditionalists – people who doubt the veracity or usefulness of many of the new approaches, who believe primary qualitative and quantitative research will always be needed, etc. Not everyone falls cleanly into one camp or the other but there are clearly lots of members in each and things are sometimes pretty ugly between the two.
For many Traditionalists, the Disrupters are often unrealistic dreamers who latch onto something largely because it’s shiny and new, not because it’s proven to be effective. For many Disrupters, Traditionalists are afraid of the future, unprepared for inevitable change and too busy sticking their heads in the sand to see where things are going.
Unfortunately, both sides have some significant gaps in the logic of their typical arguments. In order to offend as many people as possible, let’s look at some of those logic gaps.
Failing to look at history (or only looking selectively at it). Disrupters often conveniently gloss over all the failed predictions of the past. Focus groups will be extinct within a couple of years, huh? We’ve heard that prediction for at least the last 20 years – certainly since the advent of online qualitative. No one will do surveys in the very near future? Yep, heard that one too, for quite a while now.
Hydrogen-powered cars … the paperless office … supersonic travel … the shorter work week … all of these things were “inevitable” according to many pundits. I’m still waiting. Could some of the new insights techniques end up being just as “inevitable”? Which next-gen techniques will end up as the equivalent of Google Glass?
Traditionalists do this too. They tend to ignore companies like Kodak, Borders and Blockbuster that failed to adapt to change. They argue that mail surveys and mall intercepts are still used, even though their demise was predicted long ago. Yes, they’re still used, but how often? How much money is spent on mail surveys today, compared to other methodologies? Certainly not as much as in 1990 or 1970.
Preaching to (and receiving approbation from) the choir. It’s human nature to congregate with other people like ourselves. That’s why we have Steelers bars, ethnic churches and 55+ communities. But when you spend a lot of time with other Disrupters, you can easily think everyone is a Disrupter and those who aren’t are simply out of the loop.
Since it’s also human nature to seek confirmation of your own beliefs and perspectives, like-minded people tend to offer this to each other, further confirming that the “others” are simply wrong. Everyone I know understands what I’m saying, so why can’t you? Maybe it’s because “everyone I know” really means “all the people of like mind with whom I surround myself.”
Using anecdotal information as if it proves the point. As researchers, we get exasperated when the client contradicts all of the information we have with comments like, “But those two women in the qual study were ready to buy the product right now!” At the same time, many of us are ready to turn around and predict the future of our whole industry based on the fact that our own client still does a lot of focus groups or that the research director for MegaCorporation is praising the value of eye-tracking. Fine – and your personal experience, as a sample of one, proves exactly what?
I can find anecdotal examples to support virtually any position. That doesn’t mean they’re actually relevant, any more than two people in a qual study who loved the product concept means it’s going to be a success.
Promoting certain positions because this is what we want to happen. If your company does text analytics, of course you want to promote text analytics. But so many of the comments from both sides pretty obviously originate from what the commenter desires rather than a dispassionate evaluation of what’s actually happening.
For Traditionalists, it’s pretty scary to hear comments such as, “There won’t be any surveys in five years.” If that’s what your business is built on, what do you do if that prediction actually comes true? It’s only natural to attack statements like that.
At the same time, the accusation many Traditionalists level at Disrupters is that they’re primarily trying to promote their own services. And quite often, this is true – people are writing articles and presenting webinars claiming that whatever they’re selling is the approach that’s going to change the industry (and why all the competing methods are rubbish).
Even on the supposedly neutral client side, agendas are a reality of business. If the research director has convinced management to sink a bunch of money into big data, that person is naturally going to promote big data as the wave of the future. If the corporate culture tends to punish risk takers, the research director is naturally going to be more supportive of the status quo.
Assuming what corporations do is right. Both sides have pointed to what major corporations do as evidence that they’re right. Company A cut its primary research budget in order to invest in big data. Company B increased its traditional qualitative research budget. Company C no longer holds in-person focus groups. All of these actions are thrown out there as examples to “prove” a particular point.
First, mentioning these individual companies or researchers one-by-one is just more anecdotal evidence. Second, the companies cited are usually the ones with gigantic research budgets. Naturally a comment from the research director with a $50 million budget will get more attention than a comment from the person with a $500,000 budget – but there are thousands of smaller companies and organizations out there doing research that are rarely represented in these discussions of the future of the industry. Third, remember that corporate America, for all its successes, is also what brought us New Coke, the McDLT and the Pontiac Aztek. There are too many arguments of, “The consumer insights manager from Giant Corporation said…” I’m left to wonder whether that consumer insights manager may have bought Radio Shack stock, too.
Failing to be evenhanded. So many arguments by both sides point out only the negatives or positives of any given approach. Yes, big data can find correlations you would never find through surveys. But can it tell you why? And are the correlations relevant? Spurious Correlations (http://tylervigen.com/spurious-correlations), a website with the work of Tyler Vigen, demonstrates extremely strong correlations between the per capita consumption of cheese and the number of people who died by becoming tangled in their bedsheets; between U.S. crude oil imports from Norway and the number of drivers who died in a collision with a train; and between precipitation levels in Pennsylvania and the money people spend on movie tickets. Build a business case on any of those.
Virtually every methodology has advantages and disadvantages. To either hold fast to a traditional methodology because “it’s always worked in the past” or discard it simply because there is something new and shiny to replace it are equally shortsighted perspectives. This leads to the next logic gap.
Blaming the methodology for poor execution. A perfect example of this is focus groups. I’ve heard for many years how focus groups are useless because one strong personality hijacks the group or because they devolve into group-think. That’s not bad methodology – that’s bad execution. A good moderator simply does not allow that to happen by skillfully interacting with and probing the respondents.
So the next time you hear a methodology criticized, consider carefully whether the critic really has valid points about the method itself or is unhappy because it was poorly done.
Assuming that the advent of a new approach naturally means the death of another. Mobile banking was supposed to kill bank branches. Phonographs were supposed to kill live music. TV was supposed to kill movies. Cable was supposed to kill the big three networks. Satellite radio was supposed to kill broadcast radio.
I’ve been hearing for well over a decade that online qualitative and social media monitoring would wipe out focus groups. A recent industry study showed the top qualitative methodology was – wait for it – in-person focus groups. This is a top-three qualitative method for 58% of all researchers. Methods that were supposed to replace this? Webcam focus groups were at 10%; chat-based focus groups were at 8%. Yep, looks like the death of the in-person focus group to me.
Putting arbitrary timelines on things. Even when Traditionalists and Disrupters agree on the general direction something is taking, they’ll often argue about just how quickly it will come about. Increasingly, Disrupters are making predictions that major changes are coming in the next five years … in the next three years … in the next year.
To listen to some Disrupters, by this time in 2021 there won’t be a focus group or a telephone survey to be found. To listen to some Traditionalists, these things aren’t likely to happen before we colonize Mars, if not later. Unfortunately, rarely does either side provide much justification or evidence for their timeline other than their own beliefs, anecdotal evidence or often simply a desire to be provocative and get lots of comments on their blog post.
Why in the world do some people think companies, which tend to change direction much less quickly than consumers and which often have significant investment in how they do things today, will suddenly make radical changes in how they do everything in consumer insights within the space of one to three years?
At the same time, when we look at how much the business world has changed just during our own careers, why in the world do other people seem to think the consumer insights world will remain largely as it is right now throughout the coming decades? During my own career (and I’m not exactly a senior citizen), I’ve watched the fax machine go from radical new technology to a commodity to a largely outdated machine almost no one uses (although most business cards I receive still have a fax number – but that’s another article).
Making claims of when things will happen is dangerous to both sides. You predicted online qual would replace in-person qual within five years and it didn’t happen. Guess what? You just discredited the other things Disrupters have to say and gave lots of ammunition to the Traditionalists who can now point out that in-person qual is still being done despite your claims. But since your prediction is wrong, you’ve also tacitly given Traditionalists permission to ignore the fact that reliance on in-person qualitative is decreasing – again, dangerous to both sides.
If you’re a Traditionalist, maybe it’s time to ask yourself some tough questions. Am I simply afraid of change? Am I just confused by things such as automation or eye-tracking? Have I explored these things and given them a fair, open-minded evaluation or just rejected them through fear, ignorance or stubbornness? Would I have been one of those people in 1903 laughing derisively at the Oldsmobile stuck in the mud and yelling “Get a horse!”?
If you’re a Disrupter, maybe it’s also time for some hard questions. Can I actually make a link between these new methods and business improvement? Is there really hard evidence that where someone looks first on a grocery store shelf determines which brand they buy or that that sales will rise if facial coding determines respondents show happiness when watching a commercial? Have I discarded traditional methods because there are demonstrably better approaches or just because it’s not cool or progressive to suggest a set of in-person IDIs? Would I have been one of those people in 1958 trying to determine whether I would live on the moon colony or the Mars colony by 1962?
Disruption and tradition both have important roles in our industry but usually, the real truth lies somewhere in the middle.