GenAI vulnerable to prompt injection attacks

New research shows that one in 10 prompt injection atempts against GenAI systems manage to bypass basic guardrails. Their non-deterministic nature also means failed attempts can suddenly succeed, even with identical content.

AI security company Pangea ran a Prompt Injection Challenge in March this year. The month-long initiative attracted more than 800 participants from 85 countries who attempted to bypass AI security guardrails across three virtual rooms with increasing levels of difficulty.

The challenge generated nearly 330,000 prompt injection attempts using more than 300 million tokens, creating a comprehensive dataset that reveals blindspots in how organizations are currently securing their AI applications.

"This challenge has given us unprecedented visibility into real-world tactics attackers are using against AI applications today," says Oliver Friedrichs, co-founder and CEO of Pangea. "The scale and sophistication of attacks we observed reveal the vast and rapidly evolving nature of AI security threats. Defending against these threats must be a core consideration for security teams, not a checkbox or afterthought."

Challenge participants successfully manipulated LLMs to reveal sensitive information, particularly when models had access to confidential data via RAG (retrieval-augmented generation) systems or plugins. These attacks extracted internal instructions, customer data, and secrets embedded in system prompts.

LLMs with tool access presented heightened risks, as attackers embedded malicious instructions into innocent-looking inputs, causing systems to execute unauthorized actions like sending emails, modifying files, and accessing restricted functions.

Attackers also demonstrated multiple methods to bypass content safety protections, including embedded malicious prompts in external data sources and encoding harmful instructions to evade detection, resulting in generation of otherwise restricted content.

Friedrichs adds, "The industry is not paying enough attention to this risk and is underestimating its impact in many cases, playing a dangerous wait-and-see game. The rate of change and adoption in AI is astounding -- moving faster than any technology transformation in the past few decades. With organizations rapidly deploying new AI capabilities and increasing their dependence on these systems for critical operations, the security gap is widening daily. The time to get ahead of these concerns is now."

You can get the full report from the Pangea site.

Image credit: Tero Vesalainen/Dreamstime.com