Ensuring a strong foundation
Editor’s note: Jeremy Korst is partner with GBK Collective, a marketing strategy, insights and analytics consultancy.
It’s been incredible to watch the rapid adoption of generative AI across enterprises. In a span of just two years, we have seen AI impact nearly every area of business. But the rise of AI has also raised some important questions for leaders.
Your organization may be ready to embrace generative AI, but is your data? Does your team have what it needs to maximize the potential of gen AI, improving productivity and the overall quality of work, while still understanding its shortcomings? How do you ensure AI-generated insights are accurate? And what role should AI play in decision-making? We brought these questions to AI experts to explore further.
AI is leveling the playing field
Generative AI is leveling the playing field, enabling anyone to interact with and analyze data. According to GBK’s recent survey of 672 decision makers across U.S. enterprises, gen AI is helping not only to boost productivity across organizations, but also to improve data literacy, empowering leaders to create new insights even if their background isn’t in data science.
Case in point: more than half of leaders GBK surveyed told us they see more than 20 use cases for gen AI by their team in the next three years, with top applications including data analysis, researching customer and competitive insights, content creation, coding and software development, among others.
"We are seeing generative AI speed the delivery of insights, allowing people to not spend so much time on data analyses or what I would call surface level insights,” says Eric Bradlow, GBK co-founder and vice dean of AI and analytics at the Wharton School of the University of Pennsylvania. “It’s also making more technical skills such as coding easier than ever before. All of this is positive for firms, freeing up time to focus on deeper insights and strategy.”
Harnessing AI-generated insights
Gen AI is also helping teams to innovate faster, reimagine workflows and simplify time-consuming tasks. Examples include visualizing data in new ways, generating reports or dashboards. Generative AI is also great at summarizing unstructured data. For example, you can take data sets, load them up into a large language model (LLM) and reproduce fairly sophisticated analyses purely with good prompts or commands.
"Being able to take all of this unstructured data that normally lies dormant in organizations and analyze it using an LLM allows us to add it to institutional memory,” says Sonny Tambe, associate professor of operations, information and decisions at Wharton. “It lets us understand who's doing what in the organization which is important because so much knowledge in modern organizations gets lost."
However, Olivier Toubia, Glaubinger professor of business at Columbia Business School, notes that generative AI is not a heal-all for insight generation.
“There are also some inherent differences and limitations when it comes to insight generation,” says Toubia. “While I use generative AI as a quick testing ground to be able to fine-tune language or summarize data, we are not yet at a point where it can replace humans.”
Another challenge with AI-generated insights is the need to prioritize those insights and separate the signal from the noise. The last thing companies want to create is information silos with different insights being applied by different teams with no clear oversight.
“While generative AI will help deliver insights into individual employees across the organization, it’s not clear to what extent those insights transform into organization level insight,” says Lynn Wu, associate professor of operations, information and decisions at Wharton. “That is a completely different beast."
The role of data and AI continues to blur
The role of data and AI is also blurring. For example, teams that were formally siloed around data analytics, engineering or AI are now collaborating more closely, looking for opportunities to streamline and make data and AI-generated insights more actionable.
"The first wave of interest and use for generative AI centered more on the content generation side of things … using algorithms to generate content,” says Stefano Puntoni, professor of marketing at Wharton. “Now we are seeing AI become a prediction machine … we are now thinking about a lot of problems that can actually be framed as a prediction problem.”
As teams identify new prediction problems they want to solve, generative AI provides both opportunities for efficiency as well as risks. Even with a strong approach to data governance, the reality is generative AI will often produce errors, biases and hallucinations. It is imperative that leaders have robust quality control and oversight mechanisms in place with their team to mitigate potential risks and improve accuracy. And this starts with identifying the right data.
If you feed the wrong data to a large language model, “it could just lead you down the wrong path entirely,” says Toubia. “As these models become more and more powerful, the skill set is less about coding and more about designing the prompts. How do you design experiments with gen AI, while understanding the flow or data structure, the inputs and outputs.”
According to a new survey from Google Cloud, fewer than half of leaders (44%) are confident in the quality of their data – with many leaders uncertain about whether their data can fully support gen AI use cases. This underscores the importance of organizations having a strong foundation for data governance in place to safeguard customer and proprietary company data. Success also requires continual training – making sure teams are equipped with the skills needed to interpret AI output and recognize when human input is vital.
"The key to making good decisions with data is to think without data,” says Puntoni. “The more complex the data and the analytics get, the more important the role of managers is in the insights generating process.”
Generative AI's role in decision-making
As organizations consider how to apply generative AI to decision-making, it’s important to understand both the technology’s capabilities and limitations, as well as risk-reward trade-offs.
“For example, imagine your CEO or CMO at a large tech company and you need to make a major strategic decision around the future of the company or your core product or services,” says Bradlow. “How much do you trust generative AI to help inform that decision? Would you make the same strategic decision if generative AI never existed? How much of your decision should be based on generative AI's results?”
Understanding when and how to use AI in strategic decision-making includes constant evaluation and human guidance. It's not just about AI's ability to generate insights but also about the depth, accuracy and reliability of these insights.
"One of the important to think about is as the speed of insights increases, how do you organize those insights?” Wu says. “How much is noise? Which ideas are going to be implemented? Ultimately it is up to the decision-makers, but you also have an information overload problem at management level with every worker able to give better insights.”
Exploring synthetic research with generative AI
Product development is another notable application for which Toubia is experimenting with the use of generative AI in synthetic research. Here, AI-generated consumer personas can help businesses tailor products to meet nuanced market demands, enhancing both relevance and appeal.
“As an example, we're working with a customer insights group at a large company...and one of the applications that we're testing with them is being able to run surveys on gen AI,” says Toubia. "This method involves creating synthetic personas to represent a demographic and asking them a series of questions to gather insights at a fraction of the time and cost.”
The idea of "silicon consumers" as Toubia describes, is revolutionary. "What if we actually create 500 silicon consumers that are going to be personas that are going to together be representative of the population...and then basically ask the 10 questions to these 500 synthetic respondents?"
This approach promises rapid and cost-effective market research but also highlights the challenges of ensuring the AI-generated responses are grounded in realistic consumer behavior.
Despite the potential, Toubia points out the challenges encountered, especially regarding price sensitivity in AI-generated responses. "What we found was that the demand curves were much flatter with LLMs...it's kind of creating these confounding variables that are going to explain why someone would buy for a higher price." This discrepancy shows that while gen AI can offer unique insights, there are still some gaps when it comes to human reasoning and decision-making.
Why this all matters
As generative AI and large language models continue to evolve and become more advanced, it is crucial to take a holistic approach that ensures robust data governance and ongoing team development.
To successfully harness AI’s full potential, fully integrating it across enterprise applications, firms need to have not only stronger guardrails in place to understand the accuracy, influence and impact of AI generated outputs, but also a playbook for exactly how they are applying AI and where human oversight is needed.