Chatbots can be tricked into revealing company secrets

A new Dark Side of GenAI report from Immersive Labs looks at 'prompt injection' attacks, in which individuals input specific instructions to trick chatbots into revealing sensitive information, potentially exposing organizations to data leaks.

Using data gathered from a public prompt injection challenge the report finds a worrying 88 percent of participants successfully tricked the GenAI bot into giving away sensitive information in at least one level of an increasingly difficult challenge.

In addition nearly a fifth of participants (17 percent) successfully tricked the bot across all levels, underscoring the risk to organizations using GenAI bots.

"We had a public challenge last year for anybody in the general public where they could log on to the challenge and the goal of the challenge was that they had to trick the bot into revealing the password using any techniques that they came up with," says John Blythe, director of cyber psychology at Immersive Labs. "We didn't prompt them or give them any instructions beyond that it was purely in their creativity to how they might get the bot to reveal the password spanned across 10 levels which got increasingly difficult."

The report's findings show that even non-cybersecurity professionals and those unfamiliar with prompt injection attacks can leverage their creativity to trick bots, suggesting that the barrier to exploiting GenAI in the wild using prompt injection attacks may be easier than you would hope.

"I like to think that people are just natural social engineers, they often use in a lot of the techniques that social engineering uses to deceive people in phishing emails," adds Blythe. "You know, people just naturally do that. It plays on this idea that people engage with these bots in the same way they do with other people. So when they're primed to try and trick it, they just use the same techniques they might do in everyday language to persuade people and I find that quite fascinating."

You can get the full report from the Immersive Labs site.

Image credit: Elnur_/depositphotos.com