AI use within the EU

Editor’s note: Kevin Karty is CEO of Intuify, a Boston-based market research firm. 

Over the last year, we’ve experienced a radical increase in the use of AI in market research, which is likely to accelerate. Although the U.S. and other countries have been lax in regulating the use of AI, the European Union has been working on a policy for AI safety for the last few years. On December 8, 2023, the European Parliament and Council arrived at a deal that merged two different proposals. That deal has the potential to significantly impact our industry moving forward.

The regulation is quite lengthy, so in this article I’ll address key points insights leaders need to know.

AI regulation and market research 

The new EU rule is a supplement to other rules that protect individual data privacy and personal data, as well as other protective regulations like labor laws and personal protections – the existing rules all still apply.

The current draft rule applies to AI use within the EU and, to a significant extent, outside the EU. The rule both prohibits export from the EU of banned systems and the use of AI systems in third countries whose outputs are intended to be used in the EU. Thus, processing data outside the EU does not protect from fines.

Any system that uses an AI component, and could not function as intended without it, is considered an AI system. That means that if an AI model is a piece of a market research tool, then the market research tool is itself an AI system.

There are some exclusions – scientific research, open source, military – but market research doesn’t appear to qualify for any of these (though specific applications might).

Forbidden AI activities and market research

Banned: The use of certain AI systems with the objective to or the effect of materially distorting human behavior, whereby physical or psychological harms are likely to occur.

  • This implies it would impact research using tools that incorporate AI systems and that are designed to measure subliminal or subconscious thought if the research tools in any way modify those thoughts or behaviors.
  • While the ban seems to be primarily targeted against active AI systems that influence specific individuals (like social media algorithms that unconsciously influence people), it does seem to apply to AI and biometric systems (like facial or emotion recognition) that measure the effectiveness of unconscious messaging or imagery for the purpose of designing ad campaigns that then influence people.

Banned: AI systems that categorize natural persons by assigning them to specific categories, according to known or inferred sensitive or protected characteristics, are particularly intrusive, violate human dignity and hold great risk of discrimination.

  • This could imply that AI systems (including machine learning tools) designed for market segmentation could be impacted, especially if they include protected characteristics like gender and ethnicity. Practitioners should consider limiting the inputs they use to train these models. 

Banned: AI systems providing social scoring of natural persons for general purpose that may lead to discriminatory outcomes and the exclusion of certain groups.

  • This probably applies more to marketing teams, but it does suggest that AI systems that may effectively exclude or discriminate against certain groups for targeting of offers, deals, memberships or similar benefits could be at risk.
  • Additionally, fraud scoring tools are given particular mention in this context. If AI systems are used for fraud detection in surveys (and thus restrict access to rewards), specific care should be taken that these systems do not rely at all on data on protected groups.

There are a lot of other banned applications – like emotion recognition in the workplace or facial recognition in public places – but these do not seem to apply to market research.

Additional provisions for high-risk AI and foundational models (including LLMs)

Fortunately, market research does not fall into the high-risk domains identified in Article III, but many marketing research tools increasingly make extensive use of generative AI tools like LLMs and image generators. These tools are called “foundation models” and are given special consideration and requirements.

If you are using a foundation model from a third party (through an API, for example), that third party will need to provide documentation around data training sources and register with the EU. If you are building or training your own LLM (as a research supplier or company), this may apply to you as well.

The regulatory burden for foundation models could be quite high. This includes examining training data sources for bias, applying mitigations, ensuring output does not violate other EU rules and much more. Given how inherently biased LLMs are today (because they’re trained on the internet), this could be interesting. There are additional requirements around copyrights in training data and even energy usage.

A Stanford analysis of prior rules observed that none of the exiting in-market LLMs are currently compliant. We’re waiting for a new analysis on the updated rule, but based on our initial read, it seems that OpenAI and other foundation model providers will need to invest significantly to ensure their LLMs comply. Luckily, we still have time as the rule won’t go into effect until at least 2025.

Looking ahead: AI regulation and market research suppliers and clients 

At the end of the day, what does this mean for most market research suppliers and clients? A few key points stand out:

  • If you do anything in the EU, this will probably affect you.
  • If you’re building your own LLM, be prepared to do a lot of compliance work. If you’re using an API, make sure the provider is prepared to put in the work to comply with the rule.
  • Offline applications for LLMs (like coding data or summarizing findings) are probably OK but be a little cautious about using LLMs and image generators for real-time applications, like directly interacting with customers using chatbots or auto-probes. Dynamically generating content in real time for a discussion or survey could be held to a higher standard. At a minimum, the system may need to identify itself as an AI tool and take measures to avoid bias, insensitivity and any appearance of manipulating the customer. 
  • Be careful in using biometrics or AI in tools to measure the effectiveness of unconscious messaging or imagery, especially if the output is likely to be used to manipulate people’s thoughts or behaviors. Certain areas like advertising measurement may be affected, but the scope is still unclear.
  • Avoid using sensitive data on protected categories as inputs to pretty much any machine learning model – even segmentation models or fraud scoring tools.

Overall, the regulation is reasonably friendly to market researchers and probably less disruptive than prior regulations like GDPR. As we learn more, I’m hopeful that the new rule will help us leverage the best aspects of new AI tools while avoiding pitfalls.