Testing generative AI moderator prompts
Author: Paul Kingsley-Smith, Qualzy
Artificial intelligence is everywhere. It is creeping inexorably into the services we experience in our everyday lives. It seems all around us in business too – you don’t have to look far in any business publication to find a piece musing on the impact AI will have on how we work, manage and learn.
The qualitative research community has been abuzz with such speculation for some time. Of course, research professionals are usually analytical beings, so it is unsurprising that they have paid as much (if not more) attention to the risks that may be coming as to the benefits.
Service and solution providers to the qualitative research industry have also been in the thick of the AI debate, of course. The engineering mind-set, coupled with commercial considerations, have tended to steer that debate more toward opportunity – and that’s perfectly understandable too.
However, although discussion and debate and speculative musings on what may happen in the future is great fun, it’s not all that productive. At some point everyone must take action.
It has surprised the team here at Qualzy how little real activity there has been on the platform side so far. We moved pretty fast – almost as soon as ChatGPT burst onto the scene in October 2022 we started work on integration, turning our minds to how it could support qualitative researchers.
Conducting a two-week AI research experiment
We were incredibly glad to find that one of our good partners and customers, Mustard Research, was equally curious – because bringing AI into focus and figuring out its practical use to aid the qualitative research business or individual simply can’t be done purely from the technology side.
Mustard wanted to know whether AI could truly be helpful. Meanwhile, Qualzy had developed what we felt were a nifty set of AI feature options that held potential to make a researcher’s life easier, relieve the tedium of some lower-level tasks and speed up the workflow. We needed to test them.
A collaboration was born when we realized that having both research professionals and a research platform to play with, we could actually test whether and how AI moderator prompts might deliver, when put into play on a real project with real research participants.
So, Mustard built a project using Qualzy, borrowing a real client research topic from its good customer, Mitchells & Butlers – which is endlessly curious about how consumers view its Sunday lunch. Sixty participants were secured, for a two-week AI research experiment.
Generative AI was put into play in a real online community that was split into three groups. One received AI-generated follow-up questioning and engagement with their feedback. The second had a pure human engagement with a moderator. The last experienced a mixture: AI generated response suggestions for a “real” moderator to validate, tweak and apply as their judgement dictated.
We found the results hugely encouraging. We learned that our AI moderator prompts really do have the ability to help – participants responded well to these. As they did for the “normal” moderation, too. Yet, they sometimes responded even better when the AI was helped along by a human in the blended model. Powerful feedback about respondent experiences came in as they learned progressively how they had interacted with AI – and how they felt about it.
AI and marketing research: applications and responsibilities
By bringing the art and science of qualitative research into a test environment, both parties walked away with incredible insights to inform our future strategies. Not just around how AI can be applied, but the responsibilities each side has to use AI in ways that enhance offerings and enable professionals.
For qual researchers and firms, that means remaining open to the opportunities that AI offers. For developers, it means being judicious about where we apply AI in supporting the process of project design, build, moderation, analysis and reporting – never simply putting AI in because we can.
We’re not at all surprised that research professionals have been a wee bit more preoccupied with the risks of being pushed out of the process, than with how AI could help. Its ability to power data collection at scale and across borders, and transform analysis and reporting, are well known. They don’t need the specter of being marginalized in moderation and engagement.
Yet, the joint experiment really proved to us that the human researcher should feel quite safe. While AI will do an increasingly good job of engaging in human way with participants, it simply can’t replace what qualitative researchers can do.
Used well, generative AI can enable professional researchers to do what they do best. Give them a head start in providing smart suggestions, spotting inclusions and references, and generating options to choose and use.
The human researcher, with AI support, might delegate much of the mundane and instead use their brainpower on the bit that truly matters. That is understanding nuance, comprehending meaning behind comments and feeling the emotion that is driving a response.
It is up to us all how we proceed from here. The advent of AI is happening all around us, and it will change our future whatever happens. But I can assure you of one thing – Qualzy is excited and encouraged about where generative AI can take us, together.
Discover the full story of the experiment, the learnings it unlocked, and the opportunities we saw, in An Experiment in Generative AI and the Art of Qualitative Research.
About the author: Paul Kingsley-Smith has been driving growth and development at Qualzy since 2018, with a focus on building collaborative client relationships that can help drive strategic development of the platform and brand. He is fired by a belief that digital innovation can enable inspired change and human connections, as well as build quality insight. His passion for tech and the digital environment has fueled his career and influenced all aspects of his life as a customer service leader, father of three and dedicated runner.