Editor's note: Ian Cain is the director of business development at Luminoso, a Cambridge, Mass., text analytics firm.  Eli Orkin is a marketing associate at Luminoso. Jared Schrieber is CEO and co-founder of InfoScout, a San Francisco research firm.

Market researchers seem to love scores. Whether they are in the form of ratings, satisfaction or sentiment scores, market researchers draw most of their conclusions about customer loyalty and satisfaction from these scores. But what are those numbers really saying? They never truly get to the heart of what a consumer is saying about a store or a particular product. These numbers are simply approximations of perception that don’t provide a company with the information necessary to react or to make improvements that would benefit an organization’s bottom line.

Quantitative ratings expose only shallow sentiment. They fail to surface underlying drivers of emotion, expression and behavior. It is in identifying these drivers within qualitative data or open-ended feedback – and pairing that data with quantitative ratings – that enables businesses to react and to make immediate improvements that get to the root cause of customer issues.

A score does not allow a retailer to understand common themes amongst respondents at varying rating levels or provide a brand with insight into the fact that it’s their recognition, or lack thereof, which is most affecting its rating. What it takes is a healthy mix of quant data and qualitative feedback to get to the crux of any customer-perceived successes or missteps. Furthermore, this methodology allows any researcher to validate or invalidate a quantitative score.

In the industry today, too many market researchers are spending their time quantifying and qualifying products and ideas instead of using methods that help solve issues and gain deeper knowledge of the consumer.

In-the-moment surveys

InfoScout’s mobile apps incentivize over 100,000 American consumers to take pictures of their everyday shopping receipts on a longitudinal basis. To support this study, in-the-moment surveys were triggered to these panelists based upon where they shopped and the products they bought, with the goal of better understanding their decision-making along the path-to-purchase. A retailer trip survey asked shoppers to score their shopping trip and offer open-ended responses to why they chose that store along with their experience. A brand purchase survey asked consumers to rate and review the product while also exploring the extent to which its purchase was planned and/or other brands were considered.

Using Luminoso’s market research software to analyze the hundreds of thousands of unstructured consumer responses revealed the driving themes and concepts behind quantitative ratings. The analysis provided insight into consumer behavior around purchasing habits and brand loyalty and uncovered consumer perception around retail stores, brands and specific brand categories. These are all things that would be unknown by relying only on a score.

Specifically, we looked at over 14,000 unstructured consumer responses regarding experiences at over 50 different retail stores and over 270,000 consumer responses to over 50 brands in order to understand what drove consumer sentiment and store selection across ratings of satisfaction and loyalty.

Most prominent themes

In the retailer trip surveys, when responding to questions about store selection, the most prominent themes were “location,” “convenience,” “price,” “selection,” “coupons,” and “customer service,” respectively.

Our analysis of open-ended responses also provided insight into how these major drivers of consumer sentiment fluctuated on the 1-to-5 scale. “Convenience” and “location” were the most important drivers of consumer responses at every level. However, those that rated their shopping experiences very low or very high tended to emphasize “convenience” and “location,” coupled with “price” and “selection.” At a neutral rating of 3, consumers emphasized “convenience” and “location” over other topics such as “selection” and “price.”

Consumers who were less likely to recommend a specific retailer stressed “convenience” and “location” in their responses, while those who were more likely to recommend a specific store discussed “price,” “selection,” “location” and “convenience.”

Consumer conversation changed dramatically around different stores and retailers (Figure 1). Responses regarding dollar and convenience stores were primarily driven by “convenience,” “location” and “prices,” while responses to discount and club stores like Costco and Sam’s Club had virtually no association at all. According to our analysis, Whole Foods, Wegmans and Trader Joe’s all ranked within the top five retailers for discussion about “selection,” while drug stores like Walgreens, Rite Aid and CVS ranked in the bottom five.

Respondents overwhelmingly discussed “customer service” in feedback associated with Publix but there was hardly any discussion of “customer service” with large discount club stores (Figure 2). Availability of specific brand names and commentary around “loyalty” were most prominent in responses associated with Trader Joe’s, Whole Foods and BJ’s, all chains that are known for their private-label products as opposed to their selection of national brands. There was not very much association between those same themes and dollar and convenience stores.

When Walmart’s qualitative data was compared to Target’s (Figure 3), the difference between response themes was noticeable. Customers of Walmart stressed “convenience” and “location” when describing why they chose to shop at Walmart, while for Target they discussed “prices,” “selection,” “customer service,” “coupons” and “brand selection.”

In the brand purchase surveys, the most prominent themes among consumer responses were “positive taste,” “positive price,” “health,” “quality,” “brand recognition” and “negative taste,” respectively.

On a scale of 1 to 5, respondents who gave lower ratings on their product purchases discussed a lack of “brand recognition” and “negative taste.” Many consumers cited their reason for providing a low rating as not being familiar with the brand they had purchased. As ratings increased, themes such as “price,” “quality,” “taste” and “health” increased in prominence within consumer responses. When purchases were rated highly, consumers most often discussed the “quality” of their purchases, often emphasizing reliability, trust and familiarity with the brand.

Consumers were more likely to mention “price” and “quality” when describing shampoo products, diapers, and razors and less likely when describing candy and soda products. Chobani and Nature Valley (Figure 4) were the only brands that ranked within the top five for mentions of both taste and health benefits within open-ended responses.

The Head & Shoulders brand was isolated in order to discern the driving themes contained within the brand purchase survey responses (Figure 5). These driving themes were “quality,” “price,” “scent,” “results” and “symptoms,” respectively.

One of the more interesting findings was that consumers who rated the Head & Shoulders product within the middle of the scale discussed the “scent” of the product; as ratings increased, responses became increasingly driven by “symptoms,” “quality” and “results.” (Figure 6)

Open-ended feedback was also segmented and analyzed based on responses to a multiple-choice question about what the consumer would have done if their purchase had been out of stock. What were found were drivers of brand loyalty and detachment. Consumers who indicated they were loyal to the Head & Shoulders brand (Figure 7) tended to mention “results,” “quality” and “symptoms.” Consumers who indicated they were not loyal to the Head & Shoulders brand discussed product “scent,” “price” and “quality.” Potentially, hair care manufacturers looking to drive share growth should not only continue to drive quality and product results but also provide appealing scent options that capture more product switchers and differentiate their brands from any similarly competing brands.

Trends in purchasing behavior were revealed about the Head & Shoulders brand by segmenting and analyzing responses to a multiple-choice question regarding planned purchases (Figure 8). Those who planned the purchase of the product discussed “positive results” and “quality,” while consumers who made an impulse purchase were much more likely to discuss “symptoms.”

Quantify what drives variations

Scores alone are not the most accurate measurements of customer satisfaction, loyalty, experience or brand perception. In fact, scores are just the tip of the iceberg. A score is a perceived relative impression that provides limited insight into how or what a consumer is truly thinking. By incorporating qualitative data into research initiatives, it is possible not only to quantify what percentage of your customer base is pleased or displeased with your products and services but what drives variations in consumer sentiment.

Companies and market researchers should take careful consideration into how they go about deriving insight into consumers. Without a balanced methodology for taking the expressive open-ended survey responses against the structured components of a score, they might never discover the nuanced perception of their valuable consumer.