The more things change

Editor's note: Bill Bean is executive vice president, research and insight, at Shopper Intelligence USA. 

Shopper insights as a discipline has gone through a few definitions and practices in the 20 or so years since it was recognized. Much has changed in retail and marketing. New retail technologies and ways of selling crop up at every industry meeting and capture our attention every day.

The challenges presented by new channels and technology demand knowledge to manage success and many people have argued that they dictate completely new approaches to research. In the rush for the new, there are concerns that proven fundamentals may fall by the wayside. Even in complex and innovative times, we should not be so quick to reject them. They are still productive. 

Fundamental #1: It’s not just about stores

A shopper insight is learning at the point of purchase that can shape, influence and refine marketing and sales strategies. Shopper research produces those insights. There are plenty of other definitions and while it’s probably not worth spending much time dissecting or specifying your definition to perfection, having one does help an organization know what it is talking about and why it requires focus.

The point of purchase itself refers to any area where a transaction happens. Note that the definition is not specifically about a “store.” It refers to any place that is designed to capture the attention of potential buyers and convert that attention to a sale. That place should manifest the marketing and sales strategies of sellers and “store” owners. It may be a store, an e-commerce site, a community market exchange, a restaurant, a bank or even a doctor’s office. The principles of shopper insights apply to any area where a transaction takes place. I’ll refer to them all as stores for ease.

Fundamental #2: Shopper vs. consumer is an important distinction

This definition also implies a clear delineation between shoppers and consumers and between shopping and consumption. Although obviously related, the targets and their psychology are different. The focus of shopper insights on purchasing per se dictates different questions, business challenges and, sometimes, different focus. Needing a common language with retailers focuses shopper insights on categories. At the most fundamental distinction, shoppers and consumers can be entirely different people and not just have different mind-sets and goals. Pets and infants, for example, never shop for their own food and toys. Shopping and consumption are different behaviors and require different approaches.

There are two common questions about this distinction: Do I need separate people to manage shopper insights versus consumer? If I do need separate people, where should they report in the organization? 

The answer to the first question is: not necessarily. There are some variations in skillset but generally insights skills travel well between the two disciplines. So the answer really depends on the needs and culture of the organization. 

The answer to the second is: it depends. Research shows that reporting structure is not related to shopper marketing productivity. The Conference Board and several of the large management consulting firms have indicated that while lots of organization structures exist, none has an effect on productivity. Results show that any successful insights discipline must focus its priorities and, above all, be curious. With few exceptions, if the organization or individuals are curious enough, any reporting structure can be productive. 

Fundamental #3: Questions are really important

The questions generated by that curiosity are the key drivers of valuable insight. These are my big strategic shopper insights questions:

• What drives category growth? 
• What is the size of the prize?
• Who are the key category targets? 
• What are those targets’ wants and needs?
• What is the sequence of shopper experiences that lead to purchase?
• How and where can we influence that path to purchase?
• How do we optimize the reach and content of shopper communication to be more influential?
• How does the purchase environment affect expectations and outcomes?
• How do we ensure the maximum impact of point-of-sale programs on key indicators?
• How can retail and brand equity interaction be leveraged to mutual benefit?

These are enduring strategic questions, as relevant today as 20 years ago, and as relevant to new channels as old and it’s obvious that as a discipline we have not finished answering them. 

These questions cascade into everything on the shopper insight agenda and dictate the methods used to find answers. They bring life and order to curiosity and refine the ability to understand and overcome the business challenge. Any methodologies should be chosen after comparing their ability to meet the goals with the resources available. 

Getting the question right leads to deeper insights and more innovative solutions. I designed and co-teach a course on how marketing and salespeople can become more insightful. We do not teach research techniques but instead focus on asking the right kinds of questions to inspire; find potential root causes; stimulate and provoke. If you want to improve or start a shopper insights group, start with the quality of your questions, not with techniques.

Here is an inspiring list of shopper questions from Kate Newlin’s book, Shopportunity!: How to Be a Retail Revolutionary: “I want to understand why we’re willing to spend $3.50 for a latte at Starbucks but bristle at a 10-cent increase in the price of toothpaste. I want to learn why we’ll drive miles out of our way to buy a bag of 100 razor blades for 50 cents less than at our local store and then spend $3.99 on a tub of pretzels we didn’t know we needed and that we’ll resent having eaten once it’s gone.”

I’d like to know those answers too. They are inspiring shopper insights questions. They are precise, researchable, energizing and generalizable to other situations.

Fundamental #4: Existing tools for knowing shoppers are good if you know how to use them

Shopper insight tools are similar to those of other disciplines but often applied in a different way and with a different emphasis. The shopping environment plays an important role in usage and results.

I prefer this higher-level grouping of shopper insights tools: observation; conversation; measurement; experimentation; modeling.

All are important and all have demonstrably led to increases in our understanding of shopper motivations and behavior. Not every problem requires every tool but the use of every tool adds to the general knowledge of shopping.

Observation. This is the fundamental technique of shopper research, relied on more often than in any other insights discipline. Despite the oceans of scanner data or click patterns, there is no substitute for observing your shoppers. Sometimes observing is enough by itself. 

The medium is irrelevant (live or filmed). Seeing the action is essential. In our course on becoming more insightful, we show videos and simply ask, “What did you see?” Invariably, a video of a child and his father shopping for breakfast cereal will quickly bring forth product placement or POS material critiques versus a description of the activities seen. Only after multiple “tell me what you see” requests does the class realize their error. Only then do they note who the real shopper is (the child) and the limitations posed by the child’s height (can’t see top shelves). Quieting the inner expert and ego is key to really seeing and then discovering profound shopper insight.

Conversation. There are many ways of talking to shoppers (one-on-one IDIs, focus groups, projective techniques, surveys, shop-alongs). All have limitations but all are also valuable for certain shopper insight problems.

In Shopportunity, we see the author’s search for deeper truths about shopping behavior. She conducted shopper interviews to ask hypnotized respondents to describe in detail their most “memorable and powerful shopping experience.” The majority of women recounted the purchase of their wedding dress. Most men shared the purchase of their first car. 

Newlin found the ideal, super-powered shopping experiences against which all other shopping is compared. These experiences share psychological components: anticipation, pursuit, prominence and appreciation. With high levels of anticipation having built for years, the perfect item is hunted and located with some difficulty. Once attained, the purchase gains status, is treasured and gives back in memories for years. It is a powerful story whose significance is buried in the subconscious but revealed through conversation with the shopper.

The availability of massive amounts of data leads some researchers and marketers to deride these data, calling the stories lies, exaggerations or the result of faulty memories. “Shoppers don’t know or remember why they choose, so the data are unreliable at best and mendacious at worst.” They advocate passionately for the abandonment of these tools in favor of “more objective” methods. 

Tools are valuable if you know how to use them properly. A well-designed survey, with appropriate benchmarks and analytics, is still a source of insight. Used properly it will reward; otherwise, it is like using a screwdriver to drive a nail. Failure isn’t the screwdriver’s fault.

Measurement. There is no shortage of shopper-related things to measure, from stores, sales material and respondents. As usual, deciding which measures to use depends on the question as well as who is asking (buyers or sellers, retailers or vendors, creators or users). And how you will be collecting the data, whether it’s direct behavior measurements or the result of shopping. It is hard to see the value in “why” questions if the “what” and “how” are not adequately and accurately measured.

Some measures prove to be extremely useful. However, the current state of measurement in shopper marketing gives truth to the Albert Einstein quote, “Not everything that counts can be counted and not everything that can be counted counts.”

Evolving technology offers opportunities to measure in new ways. E-commerce practically defined the term “big data” and sheds light on the digital path to purchase. Similarly, the physical path to purchase can be defined with precision (previously impossible). Tracking sensors and videos measure shoppers’ interaction with products and aisles so the physical path to purchase can be defined with precision (previously impossible). 

Nielsen or IRI data will show a product selling in 80% of the stores in the country, for example, while direct measurement in the stores shows only 20% of shoppers actually see the item. Thus the effective national distribution is ~16%. 

It’s been proven that understanding and measuring behavior leads to improvements that help generate increased sales. For example, designing stores for excellent shopping experiences with accurate knowledge of behavior improves annual revenue by as much as 12%. Using your questions and decisions to guide measurement is a good way to sort the measurement potential. Is the measure valid? Is it accurate? Does it help to understand your questions?

Experimentation. There are lots of ways of knowing the marketing world but it is hard to beat a good experiment with a great control group. The above assertion about the potential of physical store redesign based on actual shopper behavior is taken from real data from an experiment. A large-format retailer in the northeastern U.S. rearranged an entire store based on measures and analysis of shopper preference and traffic. The principled redesign immediately resulted in a 17% increase in sales versus a nearby control store and a sustained 12% increase over the entire year. There was no other remodeling. 

Large e-commerce businesses conduct thousands of experiments every day, making design decisions driven by data and confirmed by trial and error. In the digital world, almost everything we experience has been chosen by experiment; by myriad shoppers choosing one option (A) over another (B). Results and adjustments are realized almost in real-time.

That approach is more expensive and tedious in the physical world but it is still important. Evolving virtual reality technology is allowing testing of almost any physical option in a virtual setting where we can represent any store environment in a virtual manifestation of the shopping experience. In the words of Stephen Needel, a pioneer in this technology, “We can conduct package tests without a package, price tests without paying and store tests without the store.”

Experimentation should be a vital component of any shopper insight practice and virtual testing is an essential shopper insight tool. For example, there is a longstanding theory in CPG space management that the shelf arrangement should reflect how shoppers make decisions. For our category the metric for the hierarchy of shopper decisions clearly indicated that shoppers were more loyal to a key product attribute than to brand. The theory would then dictate that the shelf should be arranged almost exactly the opposite of the current state, which assumes the brand is the most important. Rearranging those shelves even in a limited test would be both logistically and financially very expensive so we tested the idea in a virtual environment. The results of all of the virtual testing showed no difference between the two conditions. The virtual experiment both cast doubt on the practice based on the theory and prevented us from any expensive upheaval at the shelves.

One defends virtual shopping by using the same argument for surveys. Yes, VR is not real. Yes, respondents don’t spend real money. But despite those seeming limitations, shopping decisions in a virtual store mostly correlate with shopping in a real environment. Item shares match. Trial and repeat patterns validate. Even purchase dynamics largely agree. These validations are regularly published. 

Properly trained and prepared, virtual store shoppers provide us with valuable and valid data to guide decisions. What limitations we know are pretty easily avoided or managed. Purchase decisions involving heavy items and shopping where any sensory experience besides visual is important can be a limitation. But for the most part virtual reality shopping is a valid tool for experimentation. 

Modeling. At their best, models are ways of thinking as much as they are an approximation of reality. Whether physical or mathematical, models are a way to make sense of a complex world and shopping is a complex world. There is lots of evidence that the use of models improves decisions. In fact, evidence shows that using lots of models helps even more.

Models help us organize information, make better and faster decisions and develop and adopt more effective strategies. Given the amount of data in the shopper discipline, it should be no surprise that models are useful. However, there is no question that among all of the described ways of knowing about shoppers, model-building is the most complicated and requires much technical skill.

The model’s ability to rapidly simulate a market or environment and reveal the consequences of a change or disruption is invaluable. Pepsi built a model of workplace beverage consumption that illustrates this. The workplace shopping environment is data-poor. Beverage sales can only be captured through survey or diary methods. We modeled a 35-employee business whose office was in an office park to understand potential influences of consumption and purchase. The model development allowed for continuous comparison between our hypotheses from survey analytic work, real-world observations and the model. Model assumptions were adjusted to gain harmony among the sources. We then simulated the effect of different marketing and sales tactics on outcomes.

Prior to modeling, most research identifying characteristics of workers, demographics and consumption patterns implied that the only way to grow was to make access to beverage consumption as ubiquitous and easy as possible. Modeling simulations revealed a surprising tactic. The optimum vending strategy was to place machines in the building’s lobby, not close to the offices. As few as two machines in the lobby accounted for as much consumption and sales as four times that number on the office floor. The maximum exposure to building traffic three times a day drove more consumption and sales than the convenience of having the machine on the floor. Subsequent field-testing confirmed the strategy. One model overcame years of assumptions about the best way to deploy vending machines.

Frankly, it does not always work that way, but models should be an essential tool for shopper work.

Fundamental #5: There is long-term value in understanding

In some cases, we might prioritize accurate prediction over understanding. This is the situation for many digital A|B tests. An algorithm or a test uses the data to discover whether one alternative is better than another. The prediction may produce no insight behind the result and in the world of just-in-time digital ad delivery and algorithmic design, the prediction is often judged enough. While this process may be efficient, it is a big change from the past, where management understanding was a key to success. The need to understand is a foundational reason for market research itself.

Prediction and understanding are related but independent goals of market research and it is possible to separate them. That is a business decision. The complexity of big data and digital environments has increased the frequency with which predictive models take precedence but striking the right balance between prediction and understanding is still required for shopper insight.

While accurate and timely prediction is often essential to leveraging opportunity, understanding is as often the foundation for innovation. Ancient Babylonian mathematicians were extremely good at predicting eclipses and other astronomical phenomena but it was not until 800 years later that Ptolemy began to understand how the planets and sun functioned in the solar system. Ptolemy’s insight led to far more innovation than just knowing when an eclipse would occur. Understanding why something happens is still important.

These overall processes and questions of shopper insight endure but smaller, tighter, seemingly important and urgent questions appear, driven by perceived crises, fire drills and novelty. Despite the trend, e-commerce is still in its infancy for many categories and its knowledge base is fairly shallow. At an advisory meeting for the fast-growing e-commerce arm of a large retailer, the head engineer reported finding they were continuing to deliver ads to customers who purchased expensive products post-purchase. He announced they were discontinuing that “inefficient” process immediately. A few veterans informed him that continuing to view advertising after big-ticket purchases is common across many categories and cutting them off would hamper the store’s ability to cement loyalty and convince the shopper they had made the right decision. The engineers had no exposure to wider learning about shopper behavior and their logically optimized decision was not optimal in the broader context of marketing.

Generating much anxiety 

The current situation is forcing the accelerated adoption of online shopping tools and generating much anxiety about the collective uncertainty of the future. However, sometimes we know more than we think, if we will only stop and see. Although we will never know everything, we can use these fundamentals to ask good questions and to know enough to make good decisions.