Going beyond self-report

Editor's note: Jason Martuscello is a business strategist at New York-based professional services firm BEESY.

Every week, I go to dinner with a good friend. Like most predictable aspects of life, we eat at the same restaurant, at the same time and we always take his Audi Q5 hybrid because hybrids get privileged parking in Miami. One evening on our way to dinner, everything was going as planned until his car broke down. Putting aside my initial concerns about my empty stomach, my attention went to trying to understand what was wrong with the car. We were not out of gas and the battery had 3/4 charge so I was unsure if it was an electric motor issue or a problem with the gas engine. We were hoping something smaller and less expensive like a spark plug or battery line. 

Diagnosing my friend’s car, in many ways, resembles the marketing researcher’s role of understanding their customers. We have a business problem, we identify the cause and we implement the most cost-effective solution. 

The rise of new implicit methodologies and technologies to help understand customers and deliver innovative solutions is both exciting and overwhelming. Exciting, knowing we can solve our problems faster and cheaper, yet overwhelming, as it is impossible and impractical to keep up with the advantages and disadvantages of all the new methods and figure out what is best for our situation. Making matters worse, navigating service providers is difficult at best. With every service solution claiming to have “the answer” – to capture the subconscious, to better predict behavior and to deliver results – how can we be confident in choosing the right solution?

This article seeks to bring clarity to the much-talked about implicit measurements by offering strategic guidance for their appropriate use and application in market research.

Frame our business question 

Before digging into the subconscious and implicit methods, it’s first important to frame our business question correctly. Framing the business problem, in many ways, identifies the service, solution or method to approach the problem with. When our car breaks down we don’t start by replacing the car or the engine but instead we ask the right questions to isolate the problem. Likewise, consuming service providers without framing the question can limit the results or outcome of the research. For example, a simple framing of the business question, “Why are customers buying the competitor’s brand over ours?” vs. “Why should consumers buy our brand?” completely changes the methodological approach to take. 

Getting to the hearts and minds of consumers is simply not the job of self-report surveys. There is no doubt that these measures provide invaluable insights for market researchers while being inexpensive, accessible and easy to analyze but they are often not useful to explain why people experience what they do (e.g., do not reveal causal processes). Why do I buy Coke over Pepsi? Is it taste, price, color of the can or because my friends drink it? Many of the behavioral influences are simply beyond our awareness and cannot be obtained through traditional self-reports. 

Growing interest among researchers in going beyond self-report (asking direct questions) has spawned the proliferation of implicit measurement tools. Psychological and biological tools, from eye-tracking to reaction-time methodologies, have become popular yet remain widely misunderstood and lacking in practical payoff. Here we will highlight what researchers need to know to make rational decisions applying implicit tools.

The goal of implicit tools is to understand behavior. They provide the means to understand why we do what we do. Are customers buying out of emotion, habit or social influence? Implicit measures attempt to capture the deeper psychological causes of social perception, judgment and behavior that are not easily accessible when asked about or through introspective experience. It is worth noting that implicit measures don’t actually measure psychological causes (e.g., emotions or motivations) directly but rather infer what is happening inside our brain from people’s biological responses or performance on experimental paradigms and tasks. For example, heart rate is a good proxy for emotional arousal but it does not indicate whether the arousal is good or bad for brand-buying, since arousal can be accompanied by excitement or anger.

Takeaway: Implicit measures are indirect assessments or tools that infer mental constructs (emotion, motivations, preferences, etc.) in ways different from asking direct questions.

Two broad classes 

For the purposes of market research there are two broad classes of implicit measurement tools: attitude-based and biology-based.

Attitude-based assessments, more widely known as implicit social cognition, are indirect assessments of attitudes. The effectiveness of attitude-based implicit assessments is rooted in the associative network theory, which holds that attitudes are stored and activated in memory and activation of a mental concept can spread to other concepts. Think “what wires together, fires together.”

For example, if we are primed with the word “thirst” our brain can activate related concepts of “cold,” “tasty” and, hopefully, “.” These measures rely on associative links between concepts and their strength, which is assessed via reaction times. Although there are numerous attitude-based implicit tools, only two have demonstrated reliability (acceptable internal consistency): the implicit association test and the affect misattribution procedure.

Biology-based assessments include facial coding, eye-tracking, EEG/fMRI, galvanic skin response and heart rate and are indirect assessments based on our biology/physiology. Biology-based assessments rely on objectively measuring heart rate, eye movements, brain activations and facial movements to infer psychological states such as motivations, attention, preferences and emotions. 

For example, if we watch consumers watch a Coke advertisement, changes in their facial muscles can allow us to draw inferences about emotions and increases in heart rate can help us understand arousal and ultimately make inferences about whether they liked the advertisement and would buy the product. Each biology-based assessment differs widely in its link to behavior. 

What are we actually measuring?

Finding hidden truths, uncovering deep associations, overcoming biases and getting to the inner workings of human cognition are some of the frequently cited claims for implicit research methods. Putting the hype aside, what are we actually measuring?

If I ask you why you buy the toothpaste you do, your answer, although explicit, may likely be, “Because it’s cheap” (price). Although price may indeed be true, there are likely six other brands within 10-15 cents and five inches of shelf space to your purchase, making it highly probable that other influences are involved in the decision.

Answers to direct questions are not pure measures of conscious processing (e.g., they involve unconscious processes) but the reciprocal is also true – implicit measures are not pure measures of unconscious processes. In other words, implicit measures will involve conscious processing (although in many instances the goal is to minimize conscious processing). The widespread assumption is that implicit tools capture the subconscious and although implicit tools overcome limitations of traditional methods, they should not be thought of as pure measures of unconscious processes.

Takeaway: Implicit measures capture the less-conscious aspects of cognition.

Seamlessly integrates 

Just as hybrid cars seamlessly integrate gas and electric power, our brain seamlessly integrates conscious and non-conscious processes in executing decisions and behavior. System 1 vs. System 2 conceptualizations (e.g., non-conscious/conscious, implicit/explicit, direct/indirect, automatic/effortful, etc.), which neatly categorize mental processes into two distinct categories, are great for simplifying and communicating the complexity of cognition. However, if you are trying to understand how different cognitive and affective states influence behavior then we advise disaggregating consciousness into its component parts – control, deliberation, intention and effort. For example, a running habit may be automatic and conscious whereas the impulsive purchasing of candy at the cash register may require automatic, effortful control to avoid.

Takeaway: System 1 and System 2 operate in tandem and researchers would benefit from understanding their respective contributions. By decoupling consciousness into its component parts – control, deliberation, intention and effort – researchers can better understand the drivers of behavior in various choice contexts and situations.

Compared and contrasted

Implicit measures are often compared and contrasted against explicit measures, leading to the widespread assumption that implicit measures/tools are better than explicit self-report ones. Saying implicit is better than explicit is like saying a hammer is better than a screwdriver. Whether you are asking a direct question or using an experimental paradigm or tool to infer an answer, each method has its own unique strengths and weaknesses.

All too often, researchers and even scientists, both confidently and incorrectly, assume that implicit tools just work better. Or perhaps because “everyone else” is using implicit measures so should I. This is simply unsound science. As Philip Tetlock says, “Popularity is a poor proxy for utility.”

Takeaway: Avoid casting a halo around implicit measurements; know when they are superior. Also, avoid comparing explicit vs. implicit as we may lose sight of when to apply them and how to extract value from them. Use explicit and implicit to complement each other. 

Isolate relevant and effective methods

When my friend’s Audi broke down, we didn’t go to the dealership and replace it or drop in a new engine. Instead, we determined the problem and identified the parts and found the professional for the repair. In the market research world, our objective is no different: identify the problem and isolate relevant and effective methods to help us solve for our business situation.

In many cases, well-designed self-report surveys can fulfill a company’s information needs. Are self-reports perfect? No. Can they provide critical business insight in a timely and cost-effective manner? Yes. 

With concepts like cognitive biases and human irrationality taking center stage recently, it is worth noting that people are real and can provide accurate answers to well-designed and structured surveys. Tremendous business value can be extracted by optimizing questionnaire design, with attention to detail in structuring response formats, ordering effects, question wording and preceding questions. People who claim self-reports are unreliable sources of information typically are misusing them, do not fully understand their limitations or are trying to sell you on implicit methods. 

Here are five points to improve the validity of your self-reports. Ask if subjects reporting on their behavior can: understand the question; recall the relevant behavior; make an inference and judgment of the behavior; map their answer to the response format; and avoid editing their answers for reasons of social desirability.

Go deeper

The incremental value of implicit tools is generally to go deeper and beyond the capabilities of self-report. The two key limitations of self-report (outside of poor survey designs) are motivational distortions and lack of introspective access.

“Motivational distortions” simply means people are motivated to protect their image, please researchers or flatter themselves. We all have a social image and categories, products and questions that call into question our social image are prone to respondents editing their answers. For example, if I am a frequent Reese’s candy and Coke consumer I may not want to reveal my true preferences so that I can be seen as adhering to the healthy eating trend.

 “Lack of introspective access” is just a more scholarly-sounding way of saying, “We simply don’t know or have an answer.” We’ve all found ourselves moving around the grocery store filling our shopping carts, just buying without thinking. Of course we can ask why people buy their toilet paper but do they really know? Was it price or the Charmin bear on the packaging? The point is, people can always give an answer if you ask them a question but in many cases, they simply just don’t know and may unwittingly express attitudes they do not have. They may be out of touch with, or simply unable to know, their own mind.

Strategic use of implicit tools requires attention to not only the benefits but also the limitations of each method. A good way to think of implicit tools is as supplements to self-report as opposed to direct replacements. Taking an integrated or multi-method approach can help extract key insights while keeping costs low. Implicit measures are tools in the toolkit and, when used correctly and in the right context, can help better understand, predict and change behavior.

Ask better questions

If you want better answers, ask better questions. Scientists have been identifying methodological innovations to increase the predictive capabilities of self-reports. For example, intention stability, motivational coherence and attitude ambivalence have been shown to better predict preferences, choices and behavior.

The temporal stability of an intention is the persistence of the intention or its durability. When intentions are stable (e.g., same over time) they are resistant to change and are better predictors of behavior. Several key factors make up intentions, including attitudes, norms and perceived behavioral control. 

Motivational coherence is when these factors cohere or point in the same direction. The more coherent the basis of the intention, the stronger the intention-behavior consistency. 

Attitude ambivalence is an attitude-behavior consistency measure. Ambivalence simply refers to mixed evaluative reactions or discrepancies in one’s attitudes. Think of it like indecision, uncertainty, confusion or a form of conflict. 

Significant differences likely exist

Some argue that people are unable to verbalize what they feel. Instead of asking if people can verbalize their emotions let’s change the question and ask why they can’t, because significant differences likely exist with how people categorize and communicate their emotional experiences.

Each individual has their own unique understanding and expression of emotions to navigate the world. Some people with low emotional knowledge may report emotional experiences at an abstract level (e.g., pleasure), where others with deeper emotional knowledge can differentiate emotional experiences at a more granular level (e.g., joy, excitement, arousal, interest). Herein lies an opportunity: identify those who have low emotional knowledge (low granularity) and build untapped emotion associations tied to your brand. Think of it as if you were a vocabulary teacher and instead of identifying and teaching words to increase a student’s vocabulary you were identifying what emotions consumers do not know how to experience and start building new associations to these feelings. Simply put, help customers better connect with their emotions, show them how it feels to experience emotions they may not know how to experience. Tune them into their emotional selves.

Consider energy and ego depletion. Can you imagine making grocery-shopping decisions after a long day of grueling work as opposed to when you wake up fresh in the morning? Your shopping will likely be much different. Attitudes, emotions, memory, decisions and behavior are strongly tied to our self-regulatory resources (ego depletion). Our daily energy status is in constant flux and quantifying self-regulatory resources could increase the precision and accuracy of our measurement models. Moreover, identifying patterns in regulatory resources across a customer base can reveal new brand communication and behavior-change opportunities. 

Consider behavioral spillovers. When we make more money we generally spend more money and when we exercise we tend to increase our eating. These are known as behavioral spillovers – when we perform a behavior in one domain it results in a subsequent downstream behavior. What is interesting about behavioral spillovers is that the causal attribution of the behavior is largely unapparent to the consumer, creating an interesting opportunity to influence. To understand behavioral spillovers, we need to take a step back and view how behaviors interact and form patterns in customers’ lives across larger time spans (e.g., daily and weekly patterns). 

One interesting approach we are applying is quantifying consumers’ mental budgeting (how they mentally allocate money to subjective categories – groceries, entertainment, utilities, etc.) to understand implicit refresh periods and ultimately change purchase patterns. Quantifying behavioral spillovers is a hot new area in understanding global behavioral patterns to open up new terrain to connect consumer behaviors with our brands.

Consider context. Choices do not exist in a vacuum but are largely context-driven and highly susceptible to social influence. Attitude-based implicit assessments are generally devoid of context, whereas biology-based experiments may involve equipment that creates artificial environments. We need to start adding more behavioral realism by incorporating context into our designs to increase predictive validity and the ability to generalize results. Be creative!

Consider measuring the future to inform the present. Most research uses the past and present to passively predict future behavior whereas newer methodologies focus on quantifying the future to proactively change present behavior. With the majority of our thinking, feeling, decision-making and behaving causally moving into the future, there is significant opportunity to understand where customers want to go and lead innovative behavior-change efforts. By “future,” I am not talking about lofty projections or farsighted forecasts but rather systematically measuring implicit and explicit future cognitions that causally link to changing customer behavior.

For instance, instead of quantifying just how people feel now by reacting to a stimulus (immediate emotions) we are now measuring how consumers expect to feel (anticipatory emotions), which is a better predictor of behavior in decision-making under uncertainty, which in today’s fast-paced environment is overwhelmingly frequent. 

Shape the marketing landscape

I’ll leave with a mention of the future of measurement: anticipation. Anticipation is positioned to shape the marketing landscape in the coming years based on the groundbreaking new findings in cognitive neuroscience, highlighting its power to pull customers into the future. While organizations have long focused on quantifying and curating customer experiences and memories, which both science and intuition tell us are bound to be forgotten, quantifying and building anticipation is becoming a new competitive advantage to grow brands, acquire customers and optimize customer experiences.

Remember: In life and business our opportunities for growth lie in the future. It is time we start measuring it.