What lies beneath

Editor’s note: Dan Hill is president of Sensory Logic, a St. Paul-based research firm.

Long based on focus groups, Web surveys and interviews, traditional market research relies mainly on rational verbalization of the decision process. But recent advancements in neuroscience have revealed that human decisions are primarily driven through emotional reactions which are later justified with rational thought. The underlying message is that market research as we know it was never able to tell us the whole story. Verbal input is beneficial in message creation as it allows consumers to categorize their relationship with brands and offers. But we are now aware that the decision process relies less on conscious, rational thought and more on subconscious, emotional impulse. This makes it more important than ever to quantify and leverage the emotional reactions that consumers can’t - or won’t - tell us.

Thankfully, as understanding of the human mind has increased, various methods of measuring emotional reactions have been developed. Facial coding is one methodology that abandons the need to rely on subjective interpretation of consumer responses. By measuring universally valid facial expressions, facial coding uses verbal input as a support, instead of the foundation, for its findings. When paired with eye tracking (which measures attention and interest by determining the visual path taken when encountering a stimulus) this method has the ability to measure both what consumers attend to when considering an offer as well as how they emotionally react to and feel about it.

This article will begin by discussing various methods that have been utilized in an attempt to quantify emotional relevance. It will address shortcomings of these technologies and argue that facial coding and eye tracking are the preeminent tools for quantifying emotional buy-in. The next portion will cover advancements in neuroscientific knowledge, their implications for the market research industry and show how facial coding and eye tracking can be useful to researchers.

Numerous systems

As the research community becomes more attuned to the importance of emotion, numerous systems have been developed in an attempt to gauge emotional reaction and leverage the subconscious thoughts that make up much of communication. Two of the systems currently leading the charge are functional magnetic resonance imaging (FMRI) and ethnography. While both offer some insight into the decision process, neither manages to covers every angle.

FMRI measures hemodynamics (blood flow) in the brain to identify and quantify emotional response. A criticism of this procedure is that it offers no definitive emotional insight. Though we have learned more about the brain in the past 15 years than ever before, science has still not provided us with an accurate and precise guide for what sections of the brain are engaged when certain emotions manifest themselves. Until this occurs, the most FMRI can offer market researchers is an assessment of whether or not the brain attains an excited state. This binary system of response/no response simply does not offer the type of insight necessary to gain actionable knowledge of consumer emotional affect as we enter an age where experience and sensory cues will be important differentiators for consumers already overwhelmed with media stimulation.

Ethnography, the qualitative description of human social phenomena based on fieldwork, is another method that has received coverage. As it involves direct observation of subjects in real-world environs, the process succeeds in getting at the what: what consumers do, what path they take, what products they buy. It is not able, however, to address why they do these things or how they feel about doing them. Furthermore, it provides no quantitative way to measure emotional reaction. In successful market research, identifying how people feel about the offer is as important, if not more, than what they do. In an age of bottom lines and ROI, reliable, quantitative measures of qualitative analysis are imperative.

Facial coding

The facial action coding system (FACS) was developed by Paul Ekman and Wallace Friesen in the 1960s as a way to measure human emotional response to stimuli. The FACS is now used in psychiatry, by the FBI and CIA for security purposes, and by computer animators at production companies like DreamWorks.

By correlating facial muscle movement to specific emotions, facial coding uncovers the how by interpreting true consumer emotional response to offers. The system is also easily applied to real-life situations as all that is needed is a video camera to record the reaction of consumers.

Facial coding is based on facial muscle movements that accompany facial expression and emotional reaction. Human beings have the most facial muscles (43) of any creature on the planet. And the face is the only place on the body where muscles attach directly to skin, making each movement perceptible. The correlates between facial expression and emotion are so consistent that they have been found in across cultures and even in people born blind. The system takes the muscles in the face and groups them into action units (AU). An action unit is defined as the minimum, visible, anatomically-based muscle activity involved in the movement of the face. There are 23 identified AUs located across the face’s three regions: 1) brow and forehead; 2) eyes, eyelids, bridge of nose; and 3) cheeks, nose, mouth, chin and jaw. Facial expressions are comprised of particular action units that singly or in combination produce facial movement. FACS correlates these facial movements with the expression of seven basic emotions: sadness, anger, happiness, fear, surprise, disgust and contempt. The method is so accurate that even false vs. genuine smiles can be differentiated.

When used in consumer research applications, study participants are videotaped viewing or experiencing the tested offer. Then a second-by-second analysis of the videotape is conducted to establish which AUs are present. Full expressions last only a few seconds, on average. There are also micro-expressions, with involuntary muscle movement “leakage” that will often reveal in a fraction of a second what the subject is actually feeling. Due to the direct relationship of AUs to emotions this process is able to determine test subject emotional response without relying on the subjective interpretation of the consumer. In short, facial coding detects real, unfiltered emotional response.

Eye tracking

The ability to determine emotional buy-in (or disconnect) is only half of the equation when it comes to the task of determining offer effectiveness. Without specific and detailed insight into what parts of the offer cause certain emotional reactions and due to the lack of human ability to provide specific information on their emotional reaction, the most market research can hope to achieve is a general sense of yea or nay. Eye tracking supplies the definite information needed by measuring what part of stimuli consumers pay attention to and for how long.

The technique uses cameras to follow eye-movement patterns in regard to a stimulus. The pupil is then triangulated on using corneal reflections to determine fixation points. The process divides eye movements into fixations and saccades. Fixations are instances where the eye pauses and holds a gaze and saccades are the movement of the eye from one focal point to another. The sequence of pauses and movements is referred to as a scanpath and is a record of where a subject looked and how long they gazed at specific parts. This technology is able to specify what components of a stimulus are the most eye-catching and determine what order images, text and graphics are taken in.

By fusing the tools of facial coding and eye tracking together, researchers can determine what parts of an offer resonate positively and negatively with consumers. This permits the creation of a more emotionally on-target (and effective) offer through the modification of existing stimuli.

Most developed sense

In evolutionary history, visual acuity ensured human survival. Early man relied on visual clues to distinguish between safety and threat, friend and enemy, adequate food supply and scarcity. Sight has therefore become the most developed of our senses, accounting for as much as 80 percent of how we learn about the world around us. Vision is so sophisticated that the rods and cones in our eyes have very specific functions - making it possible to see the panoramic big-picture view as well as to home in on fine detail - all while distinguishing between 7,000,000 colors. Perhaps the human brain has developed so extensively around perceiving and interpreting images because two-thirds of communication that reaches the brain is visual in nature.

Determining what subjects focus on is fundamental to uncovering true emotional reaction. Our sense of sight allowed, and still allows, us to survive by instantly recognizing and avoiding danger or unpleasantness. It is the first doorway of fight or flight. What people see, the context in which they see it and the order it is processed in all have implications for the emotional effect of stimuli. Eye tracking allows us to document the process by which these stimuli are taken in as well as measure the duration of attention. When used in tandem with facial coding we are able to determine specific emotional reaction to stimuli as they are viewed. The output from these methods allows research to determine, through primal response mechanisms, whether an offer is deemed as a threat or a joy.

We think in images, not words

The brain has evolved in stages. Scientists refer to the brain as triune, acknowledging the three phases of brain development along with their overlapping, but separate, functions. The limbic brain, which is the center of sensory mechanisms (as well as emotional processes), first distinguished itself from the reptile brain with the advent of mammals. This section of the brain takes in information from the senses and is where dreams occur. It processes visually. The limbic brain was in operation for millennia before the learning brain (neocortex) evolved. Language, because it is a system of abstract symbols, only became possible after the development of the learning brain. We are therefore hardwired to think in images, not words.

We don’t, indeed can’t, use each step of mental processing every time we encounter a stimulus. It would simply be overload for the three-pound universe residing in our skulls. Looking at mental processing in a linear manner that begins with input and ends with rational decisions is misguided. It is the ability of our limbic brain to use visual cues as instantaneous and accurate guides that allows us to survive each day. For example, look at traffic signage worldwide. Fairly similar, these signs are able to impart valuable information to people without the use of language.

Furthermore, using verbal input as the basis for qualitative research runs into issues of context. Human beings, as social animals, must rationalize their reaction and compare against norms to decide whether or not what they want to say is acceptable. Facial coding bypasses this over-analyzed verbal input by getting true reactions that occur as the brain processes sensory cues. Eye tracking tells us what these sensory cues are.

Two different routes

The traditional scientific view held that emotion came after rational thought processing. However, research by Joseph LeDoux of the Center for Neuroscience at NYU into fear circuitry suggests the opposite. He proposes that there are, in the emotional decision-making process, two different routes that we unconsciously choose in response to external events. Incoming sensory information first gets filtered through the thalamus, the screening device for the psyche, which evaluates the input for interest and relevancy. It then goes to the hippocampus where anything of any emotional significance or reminiscent of familiar associations gets in; information deemed worthless never gets routed to either the conscious, learning brain or the sensory/emotion-based limbic brain.

Sensory input is routed to one of two paths:

  • The limbic path goes from the thalamus straight to the amygdala. It is more immediate, responding to sensory input prior to any conscious thought. If the sensory input has enough emotional kick to it, the limbic brain and the body work together, on instinct. They ensure that chemicals are secreted to heighten our alertness while the muscles are prepared for action.
  • In contrast, on the learning path the rational brain predominates. The sensory input filtered by the hippocampus and handled by the thalamus is then passed to the learning brain. There, it gets more “rationally” analyzed - and will get passed on to the amygdala if the input has any emotional importance.

Both paths lead to the amygdala - the brain’s emotional thermometer, mobilizer and short-term memory storehouse. The amygdala instructs the body to marshal its emotional resources and prepare for response. Sensory input channeled to the low road due to emotional urgency may activate a response even before the high-road learning brain has had a chance to perform its analysis. On the other hand, even information processed by the frontal cortex and learning brain must go through the amygdala for action to be taken. No matter what path is chosen the outcome is emotion-driven action.

This is probably the most important issue that will affect how future research is conducted. Current knowledge about the decision process has demonstrated that emotions significantly drive outcomes. Conversely, current research methods ask subjects to reverse the true nature of the thought process by asking them to rationalize (think) what they feel when in reality humans feel and then think. Emotion is the driving force behind decisions no matter which path the brain uses. Eye tracking defines whether or not the correct message was delivered in the first place. Facial coding quantifies these emotional drivers and ascertains whether the intended message evokes the desired response. Together they offer the ability to create more effective emotional connections that leverage either path of the decision process by ensuring emotional importance and relevance.

The subconscious dominates

As Gerald Zaltman discussed in his book How Customers Think, cognitive scientists estimate that at least 95 percent of our thought processes aren’t fully conscious. Conscious thought is merely the tip of the iceberg. We are, therefore, much less in control of our decision-making than we believe. For marketers, this means that appealing to consumers on a conscious, rational basis and asking them to evaluate features, attributes and benefits is largely incorrect. To reach consumers and turn them into loyal buyers you must appeal to their emotions. For researchers, it means that self-report scores such as those used in Internet surveys, clipboard questionnaires and the like are not reliable.

The issue at hand in current research methodology is asking subjects to consciously verbalize their internal thoughts and feelings. But if the subconscious controls most of thought it is technically impossible for most consumers to give accurate assessments of their thoughts, feelings and reactions; they simply don’t have access to the underlying cause and motivation behind their decisions. The main implication for research is that consumers simply can’t think what they feel; they can only feel what they feel. By avoiding complete reliance on verbal responses which aren’t necessarily relevant to internal feelings or thoughts, facial coding can deliver accurate measurement of true emotional response. It provides quantitative measures of a qualitative subject by simultaneously uncovering how a consumer feels and measuring the frequency of emotion present. Eye tracking is able to mirror the subconscious journey that is taken visually. When the two methods are combined they provide emotional insights that sync up with actual visualization patterns. In other words, no longer do you have to rely on subjects claiming a certain thing caused their reaction; it is possible to know specifically what the catalyst was.

Two-thirds of communication is non-verbal. Most communication experts agree on this statistic - a percentage that jumps to 90 percent when the topic has some emotional weight to it. Humans developed the capacity for verbal communication relatively late in the game. The capacity for word-based communication came with the development of the neocortex - the most recently evolved part of the brain - which is capable of processing abstract thought. Until then we relied on sensory signals to assess and gestures to communicate. According to Edward T. Hall in The Silent Language, there are 10 primary modes that humans use to communicate, and only part of one of those modes involves actual verbal language. Verbal communication is routinely influenced by non-verbal signals we deem to be more credible.

Not providing what it promises

By not taking non-verbal response into account, most current market research methods disregard almost two-thirds of human communicative potential. By focusing mainly on verbal and - as neuroscientific advancements have shown - not particularly reliable means of communication, research that is supposed to be qualitative in nature is not providing what it promises. Facial coding leverages this often unaccounted for non-verbal communication to create qualitative data that needs no support from subjects but instead relies on their internal desires and feelings.