By Rick Kieser, Ascribe CEO

Market research has been around for about 100 years. Some might say we are set in our ways, but in reality, the industry has been rapidly innovating in response to marketing changes, technology and other challenges. We've come far from door-to-door interviewers and “Mad Men”-esque focus groups! And we are now experiencing one of the most fundamental shifts in our industry – specifically in text analysis – in decades.

An erupting volcano

The potential of unstructured data 

In the market research landscape, our traditional data analysis approach resembles a long-dormant volcano getting ready to erupt. For years, many have focused on the top of the volcano, the visible summit – the structured data that is easy to access and straightforward to analyze from surveys. However, a repository of unstructured data remains largely untouched beneath the surface. These text comments hold valuable insights that have been challenging to access and analyze. While difficult to explore, this unstructured data contains incredible potential if it could be unleashed.

As we all know, the landscape is changing rapidly. Loaded with open-end responses and unstructured data from surveys, social media and the internet, the volcano is exploding. According to Gartner, unstructured data constitutes 80% to 90% of all new enterprise data and is growing rapidly. Just as molten lava transforms the surrounding landscape, this eruption of unstructured data, which is now overwhelming for many, will similarly reshape our approach to understanding consumers. Harnessing the insights from this eruption has been one of the most significant disruptions in the evolution of market research since we began.

Many businesses are unprepared for this shift. Only about 10% of unstructured data is currently stored, and even less is analyzed, according to the International Data Corporation. Historically, analyzing unstructured data was a time-consuming process, akin to decoding hieroglyphs. Few marketing people had the time, resources or motivation for such an effort.

Is your company experiencing this volcanic eruption of open-end comments coming from customers, partners and employees? Are you ready to dig below the easy-to-analyze closed-end data to get beneath the surface to find true insights, thoughts, desires and intentions? A fast, cost-effective and proven analysis approach is the key to unlocking the potential of unstructured data.

Enter generative AI, heralding the next evolution of market research. Leveraging gen AI with text analytics to analyze this massive amount of unstructured data provides access to the insights hiding underneath the volcano. It's a transformation from the elementary knowledge of the known structured data landscape to a new universe of depth and clarity of consumer understanding. While much of the uncovered insights will be used to answer everyday business questions, if analyzed properly, the data has the reliability and strength to guide even the most critical strategic decisions for your business.  

Gen AI's rapid evolution creates challenges

Gen AI is advancing at warp speed, generating constant evolution across the MR industry. And with that evolution comes challenges. Remember that less than a year ago, the market research industry was vilifying gen AI as the industry's demise! The MR industry is now embracing gen AI to unleash the value of the huge mass of previously neglected verbatims and unstructured feedback. Many new, young firms are jumping in to sell their gen AI solutions to marketers, and to the less experienced, their solutions appear magical at first glance. As new versions of gen AI are being rapidly released, those platform developers are having difficulties keeping up with providing human-like, theme-based insights that are quantifiable and verifiable to the unstructured data analysis. Some of the challenges created by the more recent gen AI releases include:

  • Gen AI analysis creates a plethora of results – too many ideas and codes, making it challenging for brand marketers to sort out the most important ideas. 
  • Gen AI is now creating complex codes, combining two ideas into one. For example, "too salty and spicy," combines two ideas about how a food tastes. However, this result makes it difficult to determine how big an issue "salty" is and how big an issue "spicy" is. For ideas to be actionable, we need concise, singular ideas, each of which is quantified.
  • Summaries created by gen AI are not precise enough for use in decision-making. Today, marketers are enamored with the summarizing capabilities of gen AI. If you have purchased anything online lately, you can quickly find a consumer ratings summary created by gen AI. However, these summaries are often too broad to be meaningful, and it is always challenging to identify the underlying data the summary was sourced from. The summaries miss important insights and may not represent information in the right proportions. Finally, it is well recognized that gen AI may also make things up (now referred to as hallucinations or mirages) depending on how the tool was trained. 

Ascribe leverages gen AI to develop innovative solutions

Ascribe has been innovating to solve the issues created by gen AI with the latest release of one of its most impactful technology developments – Theme Extractor. Helping Ascribe stay one step ahead of the industry, Theme Extractor is included in all of Ascribe's solutions, including CX Inspector for text analytics and Coder, a verbatim analysis platform for market research companies. Ascribe has a 25-year history as the original verbatim analytics platform for the MR industry, building state-of-the-art open-end analysis solutions. We have processed more than 4 billion unstructured comments, exponentially more than all other providers' experience in processing customer feedback combined. Our developers have deep experience with gen AI and are uniquely equipped to build solutions that meet the needs of our partners.

Theme Extractor extracts superior, well-developed ideas from open-end comments

The initial version of Theme Extractor leveraged generative AI to transform the results of open-end analysis from single-word codes to descriptive, meaningful codes that articulate the essence of the idea, a huge advancement for the market research industry. Note the example below from customer satisfaction results for a retailer; whereas before an idea might be a single-word topic such as "items," with Theme Extractor, the idea becomes "have popular items in stock." Similarly, a code of "employees" becomes "more employees would be helpful." As you can see, Theme Extractor extracts much more detailed information from the customer responses, giving you a deeper understanding of the consumers' thoughts and feelings.  

The most recent release of Theme Extractor has improved the accuracy of the results and addresses the issues of too many codes and complex codes being created by the latest versions of gen AI. Theme Extractor creates concise ideas focused on one theme, correcting the tendency of gen AI to combine ideas into complex topics. As such, in a mascara product study, the complex code "Lengthens and separates lashes" Theme Extractor separates into two themes, "Lengthens lashes" and "Separates lashes." Separating ideas is important in the analysis to make the results useful for decision-making. This is a vital detail that can easily be overlooked in a sales demonstration but is critical to the experienced brand marketer or market researcher in real life. 

Also, Theme Extractor reduces the overlap between codes, thereby reducing the number of themes. In the same mascara study, the "high price" and "too expensive" codes are more likely to be one combined idea, resulting in less overlap and more effective analysis. Finally, it is important to note that during analysis with Theme Extractor, the user can suggest the number of codes to classify the results into, which further puts the power of AI under the user's control. 

Another important Ascribe innovation is the ability to quantify the emotions and empathy Gen AI identifies around a topic. Emotion and empathy can be insightful, but if they are unquantified, they are insufficient to be helpful to brand marketers. The magnitude of those emotions (or topics) and how they link to satisfaction, dissatisfaction, loyalty and other customer states must be quantified to use the insights identified. For example, an unquantified analysis of emotion might yield two ideas, "love the service" and "wish the front desk staff were friendlier," from which you would conclude that "love the service" is more important as it is a stronger emotion. However, if the results indicated that only one person said, "I love the service," and 100 people said, "I wish the front desk staff were friendlier," the latter becomes more important. The ability to understand the emotion must be combined with frequency and quantification to get a useful insight, giving brand marketers the understanding they need for actionability and decision-making.

Enabling human guidance of gen AI produces powerful customized results

Other innovations in the most recent Theme Extractor upgrade enable human control to guide and oversee the analysis and output of the results. Many users, especially experienced market researchers, want a deliverable similar to what they have produced in the past, for example, to enable tracking against previous studies or to use language relevant to the business. Ascribe's platforms enable manual input, giving you control over the automation in real-time and letting you adjust the results to meet your business needs. Some of the manual changes available in the latest Theme Extractor: 

  • Turn gen AI on or off.
  • Provide context for the data for more accurate results.
  • Easily edit your results by renaming, combining, drag-and-drop manipulation, etc.
  • Train a codebook and save it for future use (great for trackers).
  • Set the number of codes and levels of nets appropriate for your needs. 
  • Connect the theme to the original responses (drill down.)

Ascribe's innovative Theme Extractor enables market researchers to analyze a data set with open-end comments in minutes or hours, adjusting the amount of manual editing to control costs, timing and end results.

Ascribe innovations for 2024

Being the original verbatim analytics platform, we at Ascribe are continually exploring ways to harness advanced technologies, including gen AI, to develop superior open-end analysis solutions that are easy to manage and maximize efficiencies in meeting your business needs. We are building APIs to enable direct access to our system for real-time integration with other platforms. We are developing a gen AI tool to summarize results from the analysis accurately and will also provide updated charting and visualization capabilities to meet your analysis and reporting needs.  

Choosing the right gen AI approach

Using gen AI will be critical to tap into the value delivered through successful analysis of unstructured data. It is essential that you choose the right gen AI approach for you. If you seek a partner to help you get value from your text analysis, choose that partner carefully. Look for a trusted partner with years of experience in MR, extensive experience with gen AI, and live training and support services to help you when needed. When evaluating the platform, request a live demo using your data to see firsthand if the results meet your business needs. Finally, ensure the interface is user friendly and integrates easily with your current operating processes and management tools.

Recently, we have had discussions with companies attempting to build their own analysis platforms. While their initial solution might work for one specific use case, it becomes difficult and expensive when they need to scale it and make it repeatable, and even more so as they will need to continually absorb and respond to the rapid advancements in gen AI. AI expertise is scarce and expensive, so building your own platform quickly becomes a costly and time-consuming strategic commitment for your company. We welcome the opportunity to discuss your business needs and share how Ascribe can meet those needs with speed and cost efficiencies.

Gen AI is no longer a question – it's a necessity

As you work to harness the explosive insight power of open ends flooding your company from disparate sources, you must decide how to implement gen AI in text analysis so that it becomes more effective, faster and less costly. Your decision about the partner you choose to help you with this could pay huge dividends in the future. After all, it's no longer a question of if you should implement Gen AI; it's only a question of how.

Remember to look beyond the demo to your partner's expertise in gen AI, text analysis and the MR industry. Their experience in the industry will ensure that they are not only keeping up with the rapidly evolving status of gen AI but also how to translate that evolution into products and innovations that can best serve you and your business.

Contact our team at Ascribe to discuss your business needs!


About the author:

Rick Kieser is CEO of Ascribe, a market leading provider of online software solutions for open-end analysis. Kieser has 25 years of experience leading companies in the text analytics space. During his tenure at Ascribe, the company has developed an integrated suite of leading text analytics solutions (generative AI and semi-automated technologies) which analyze open ends to meet the business needs of market research firms, corporate researchers and insights teams.