Don’t jump to conclusions
Editor's note: Terry Grapentine is an independent marketing research consultant based in Ankeny, Iowa.
In our recent book, Critical Thinking for Marketers: Learn How to Think, Not What to Think, my co-authors David Dwight, David Soorholtz and I focused on helping marketers reason more clearly but the book’s topics are equally useful to marketing researchers. In particular, researchers’ application of critical thinking skills not only helps them reason more clearly about marketing research issues but places them in a more valuable position to help the marketers they serve make better marketing decisions.
In part, the book describes 60 logical fallacies, which are errors in reasoning, in the context of real-world vignettes, drawn from our own experiences. The discussion below presents three such vignettes: Affirming the Consequent, the Ludic Fallacy and Appeal to Possibility.
Affirming the Consequent
Location: Off-site meeting between HR and the sales department.
Issue: HR and sales force management are reviewing the results of a sales-training system the company has had in place for 18 months.
Tom (human resources VP): “My department has reviewed the sales force’s performance before and after we deployed the new training system. Sales are up and I, for one, would attribute that to the new training system. What do you think, Brian?”
Brian (sales VP): “Well, we spent a lot of time screening different training system companies and it seems we selected the right one. I vote that we continue with the program.”
Tom: “I agree, Brian.”
Tom and Brian are assuming that the improved performance of the sales force can be attributed to the training program and their inference seems reasonable. If the sales training program is effective and all other factors are held constant, sales would increase. Sales increased; therefore, the sales training program has been effective. This conclusion may be correct but the logic isn’t.
Definition: Affirming the Consequent is a type of argument that takes the following form:
Premise: If A is true, then B is true.
Premise: B is true.
Conclusion: Therefore, A is true.
In the above vignette, Tom’s argument takes this form:
Premise: (A) If the sales-training system is effective, then (B) sales will increase.
Premise: Sales increased (B is true).
Conclusion: Therefore, the training system is effective (A is true).
However, Tom’s argument is not valid. Validity in this context means that, if his argument’s premises are true, his conclusion is guaranteed to be true. The classic example of this kind of validity, which you probably first came across in college, is demonstrated in the following argument:
Premise: All men are mortal.
Premise: Socrates is a man.
Conclusion: Therefore, Socrates is mortal.
In this example, the premises are true and, as such, they logically guarantee the conclusion to be true – if Socrates is a man and all men are mortal, by definition, Socrates is mortal. In contrast, the logical structure of Tom’s argument simply does not guarantee his conclusion to be true because other factors may have caused the sales increase. “If the sales training system is effective, then sales will increase” is Tom’s premise but it is not a truism like “All men are mortal.” Tom set it out as a true premise but that does not mean it is true. Training is not guaranteed to achieve the results; it might and it should but it is not a certainty. Sales and training have a more complex relationship than death and taxes.
Discussion: Sales might have increased, for instance, because of an improving economy, mistakes made by competitors, changing consumer tastes or the simple fact that the sales force is 18 months older and more experienced. Of course, you might find this to be a trivial logical fallacy or one that is self-evident. After all, we all know that “correlation is not causation.” Nevertheless, training programs are expensive and management should require better justification to approve such expenditures.
Tom should be looking for multiple, empirically-based indicators that support the belief that the sales training program is working. He should not just focus on the single metric of sales volume if he wants to make a strong argument supporting his conclusion.
Think of it this way: If the sales-training system is working, what else should be true? Example empirical indicators that could corroborate Tom’s claim might be the following: After the sales-training program, (1) the percentage of initial prospecting calls that result in a sale increases; (2) reps follow up on customers’ inquiries quicker; (3) there are fewer customer complaints; and (4) customers express a higher level of satisfaction with the sales force’s performance. The more relevant evidence Tom can produce to support his claim, the stronger his argument will be that the training program actually affected sales positively.
An important lesson for marketing researchers in this regard is not to let marketing managers believe that a marketing program’s successful outcome means that the program was well-designed. Other factors could have played a role, which marketing research can shed light on. In this regard, one of the more famous examples often cited in marketing research textbooks, in chapters on experimental design, is called the Hawthorne Effect. “The term was coined in 1958 by Henry A. Landsberger when analyzing earlier experiments from 1924–32 at the Hawthorne Works [a Western Electric factory outside Chicago]. The Hawthorne Works had commissioned a study to see if their workers would become more productive in higher or lower levels of light. The workers’ productivity seemed to improve when changes were made and slumped when the study ended.”1 Subsequent research indicated that it was not the varying light levels that affected worker productivity but the simple fact that management was paying attention to its workers.
Dealing with Affirming the Consequent: When a colleague confronts you with an argument that affirms the consequent, employ the following two strategies. First, simply point out that there are potentially multiple causes of B – the consequent of whatever A is. Second, help your logic-challenged friend think through the following question: If A is true, what else should be true in addition to B? If you can’t think of anything, then maybe A is false.
Remember, antecedents don’t necessarily guarantee consequents. Correlation is not causation.
The Ludic Fallacy
Location: Pharmaceutical marketing “war room.”
Issue: Planning the advertising rollout of an over-the-counter flu remedy prior to the fall flu season.
Kevin (senior big data analyst): “In one of our recent big data projects, we examined tracking the word ‘flu’ in Google’s search engine and found a high correlation between people doing Google searches on the word ‘flu’ and retail sales.”
Laura (VP sales): “Wow, Kevin, that’s an amazing result. Think about how we can use that information to manage inventory at the regional level, as well as to fine-tune our Internet ads and POS.”
Raechel (logistics analyst): “That’s a great idea. Our team can work with Kevin’s and develop an inventory and distribution strategy for the next flu season.”
Laura: “And I’ll schedule a meeting with our digital group to work with them on honing our digital strategy.”
This invented and abridged conversation certainly seems plausible, given that it is partly true. Google did publish an article in one of the top scientific journals, Nature, describing how accurately “Google Flu Trends” tracked the spread of that virus in 2008.2 So, about now you might be asking, “Where is the fallacy here?” It emerges when we look at what happened a year after those data were analyzed, when the U.S. flu season seems to have confounded Google’s algorithms.
Definition: Google fell victim to the Ludic Fallacy, in which highly precise (notice, we say “precise” not “accurate”) statistical and probabilistic models fail to model the nuances of the real world. Google’s estimate for the number of cases during the 2009 Christmas peak of national flu season was almost double that of the CDC and some of its state data showed even larger discrepancies.
Discussion: Ludic comes from ludus, a Latin word for game and was popularized by Nassim Nicholas Taleb in his 2007 book, The Black Swan. In his book, Taleb contends that complex statistical models and algorithms – the outputs of which are often used to support marketing arguments – are inherently biased because of the following factors:
- all relevant information about a topic is not in the possession of the statistician or the decision maker;
- small changes in the premises supporting a forecast may have large implications in the forecast; and,
- these models cannot take into account events that have never happened – for example, the mortgage crisis of 2008.
These factors are especially relevant to marketers (and it turns out, pollsters) who rely on marketing research that incorporates high-level statistical analyses – especially in regard to modeling consumers’ anticipated purchasing (or voting) behavior. With respect to these studies, you should realize the following:
- Researchers cannot measure all facets of all consumer beliefs that affect brand choice. Human belief systems are simply too complex. “Not everything that counts can be counted” is a quote often attributed to Albert Einstein but most likely originated from a book written by William Bruce Cameron.3
- There are many factors surrounding consumer behavior that are impossible to measure. For example, how one’s competitors will respond to a marketing effort.
- What is measured in marketing research studies is not measured without error. For example, survey respondents are notorious for their poor top-of-mind recall in reporting past brand purchasing behaviors.
And, to underscore the major weaknesses of highly precise statistical models is Taleb’s black swan – the unforeseeable event (i.e., it was once believed that all swans were white, until a black swan – an unforeseeable event – was discovered in Australia).
Dealing with the Ludic Fallacy: How does one, then, ensure that the Ludic Fallacy does not infect one’s marketing arguments? We have several suggestions:
First, all marketing research findings need to be combined with management’s background knowledge. Ask the question, “Are the research findings logically consistent with everything we know about this subject matter?” If not, investigate. Don’t let the numbers do the thinking for you.
Second, brainstorm alternative scenarios regarding a marketing effort, based on different assumptions. For example, when launching a new product, consider different launch scenarios based on different competitive responses or changes to the economy. The goal is not necessarily to uncover these black swans as it is to be flexible and adaptive as you determine that not all of the assumptions on which the launch plan is built are actually correct.
Third, as we’ve discussed in previous vignettes, don’t confuse correlation with causation in any big data or marketing research analysis. If you’re not sure or misinterpret what is causing the correlation between two factors, then you don’t really understand what can derail that correlation.
Regarding the last point, in examining customer satisfaction scores and sales for a major power sports manufacturer, management discovered a positive correlation between sales and customer satisfaction, which management attributed solely to their product’s quality. Over time, as the company grew, these satisfaction scores declined although product quality – as measured by metrics such as warranty claims – stayed constant. The problem was that, as the company got larger and took on more independent distributors, the quality of the distributors’ customer service declined. This was not discovered until some damage had been done to the manufacturer’s brand equity. Had managers better understood what was causing the initial high customer satisfaction scores, they could have been more proactive in screening and managing the independent dealer network.
Again, don’t let the numbers do your thinking.
Appeal to Possibility
Location: “War room” of a national consumer products manufacturer and marketer.
Issue: Brand strategists are debating plans for increasing the market share of one of their shampoo brands. To increase share, should they focus on increasing purchase frequency or market penetration?
Conner (brand manager): “The market our shampoo brand competes in is ultra-competitive. So Jackie, I just think we’ll be better off trying to increase our brand’s market share by getting our customers to use our brand slightly more frequently rather than trying to steal brand share from our competitors.”
Jackie (CMO): “Either way, Conner, it’s not going to be easy. Certainly, if we try to steal share, we’re going to have to increase our ad budget.”
Conner: “Regardless of the strategy, we’ll need to spend more on advertising. Right now I’m thinking about the strategy for the campaign. I think we need a campaign to drive frequency rather than win new customers. The Nielsen data show that the customers of the larger market-share brands purchase their brands slightly more frequently than do our customers. So it is possible to drive frequency. And if it’s possible, I’m sure we can pull it off.”
Jackie: “Well, I can’t think of any reason why your suggestion won’t work. OK, let’s develop a campaign aimed at frequency.”
Just because something is possible doesn’t mean that it will be true. Moreover, “possible” is a vague term and begs the question, “How possible?” Future events with probabilities of occurring 1 percent and 99 percent are both “possible,” yet the former is not likely to happen, whereas the latter is more probable.
Definition: Appeal to Possibility occurs when someone asserts that if X is possible, then X is likely to be true. But an Appeal to Possibility often appeals to other logical fallacies, directly or implicitly, for its justification.
Discussion: In the above example, Conner links his Appeal to Possibility to Nielsen data showing that larger market-share brands are associated with higher purchasing frequencies. His stated premise is that increasing purchase frequency is something that the larger brands do through advertising and concludes that it is something they can replicate purely because it is possible.
First, Connor is correct is claiming that higher market-share products have somewhat higher purchase frequencies.4 However he’s incorrect in his direction of causation (the large brands advertised their way to frequency of use) and, thus, has inadvertently used the correlation-is-causation argument as justification for his Appeal to Possibility.
The interesting facts that Conner does not know (and have nothing to do with this fallacy) are that, generally, as market share increases so does “mental and physical availability” of the product. Compared to small-share brands, larger-share brands have higher consumer awareness and physical distribution, causing a kind of selection effect in which larger market-share brands simply capture relatively more heavy users than smaller-share brands do.5 So causality goes the other way – increasing frequency of product usage does not cause market share growth but market share growth attracts heavier brand users.
In addition to often invoking a false correlation-causation relationship, Appeal to Possibility often joins ranks with Appeal to Ignorance for its justification – “Well, Action X is possible, and no evidence exists that refutes my claim!” But lack of evidence is not evidence supporting a proposition.
The same is true when combining an Appeal to Possibility with an Appeal to Novelty. You often hear Appeal to Novelty when you go to a marketing research conference. Someone is giving a talk about the latest and greatest bells and whistles. Companies latch onto them because of their novelty and possibility. What consultant Annie Zelm calls the “grapefruit diet strategy” is one of these fads, in which a firm focuses on a single strategy – say social media – versus integrating multiple strategies to ensure product success. In general, there is an attempt to simplify, and this is how appeals to possibility and novelty combine – it is simple (and romantic) to assume that the novel approach is working by itself, not that it is part of a more complex mix. Without disciplined diligence to find the root cause through causal analysis, marketers often link the desire to believe in the novel with the desire to believe in possibilities and come up with a fashionable me-too strategy that most likely will not work. As discussed by Zelm:
“The grapefruit diet has been around for decades but it seems we still haven’t learned we can’t live on citrus fruit alone. Nor should marketers rely entirely on a single marketing strategy while neglecting others. Some businesses lean so heavily on social media that they’ve significantly reduced their efforts in inbound marketing, public relations, e-mail marketing, direct marketing and traditional advertising.
“Social media offers the obvious advantages of instant communication and direct engagement, so it’s hardly a passing fad. That said, it’s a mistake to assume your 50,000 Facebook ‘likes’ are 50,000 likely clients or that your target audience will find you on Twitter without being prompted to look. Social media is important but, like all other tactics, it’s just one element of a balanced marketing strategy. To be successful, you need to engage your audience across multiple channels online, in person and on paper.”6
Dealing with Appeal to Possibility: Here are two thoughts on how to deal with Appeal to Possibility:
- Don’t use the word possible; use the word probable. In most cases, this is the more correct term. Most things are possible but not everything is probable.
- Investigate tacit assumptions. Does an Appeal to Possibility implicitly assume another logical fallacy for its justification (e.g., Appeal to Ignorance or Appeal to Novelty)? Perform some due diligence of the causal link.
Failed in one fundamental way
As many as 95 percent of new product introductions fail, according to AcuPoll, a Cincinnati research firm.7 Do you remember New Coke, Coors Rocky Mountain Spring Water, Kellogg’s Breakfast Mates, McDonald’s Arch Deluxe and HP’s TouchPad? Yet, during this same period, we’ve seen an ever-growing number of marketing conferences, seminars and books dispensing advice to marketers on “how to be successful.” A recent Google search of “marketing success” turned up 296 million references!
How do you explain this paradox? We believe that most of these efforts to improve business performance have failed in one fundamental way: Although many organizations and consultants provide excellent advice on what to think when formulating marketing strategies and tactics, they have failed to help today’s marketer and marketing researcher know how to think about these issues.
By constantly working to develop and improve your critical-thinking skills, you’ll avoid falling victim to the fallacies outlined above and derive the most value from the insights you gather for your company or organization.
References
1Wikipedia, https://en.wikipedia.org/wiki/Hawthorne_effect
2Ginsberg, J., et al. (2009). “Detecting influenza epidemics using search engine query data.” Nature. Retrieved from: http://www.nature.com/nature/journal/v457/n7232/full/nature07634.html
3O’Toole, G. (2010). Not everything that counts can be counted. The Quote Investigator. Retrieved from: //quoteinvestigator.com/2010/05/26/everything-counts-einstein/
4Sharp, B. (2010). How Brands Grow: What Marketers Don’t Know. Chapter 2, Victoria, Australia: Oxford University Press.
5Private correspondence between Byron Sharp and Terry Grapentine. Sharp also discusses this topic on pages 112-113 of his book (2013), Marketing: Theory, Evidence, Practice.
6Zelm, A. (2014). “Are these 4 marketing fads dragging your strategy down?” Retrieved from: http://www.kunocreative.com/blog/bid/89613/Are-These-4-Marketing-Fads-Dragging-Your-Strategy-Down
7Burkitt, Laurie and Bruno K. (2010). “New, improved, and failed.” Forbes.com on NBCNews.com. Retrieved from: http://www.nbcnews.com/id/36005036/ns/business-forbes_com/t/new-improved-failed/#.V1Xww_krLIU