From a hacker's cheat sheet to malware… to bio weapons? ChatGPT is easily abused, and that's a big problem

There's probably no one who hasn't heard of ChatGPT, an AI-powered chatbot that can generate human-like responses to text prompts. While it's not without its flaws, ChatGPT is scarily good at being a jack-of-all-trades: it can write software, a film script and everything in between. ChatGPT was built on top of GPT-3.5, OpenAI’s large language model, which was the most advanced at the time of the chatbot's release last November.

Fast forward to March, and OpenAI unveiled GPT-4, an upgrade to GPT-3.5. The new language model is larger and more versatile than its predecessor. Although its capabilities have yet to be fully explored, it is already showing great promise. For example, GPT-4 can suggest new compounds, potentially aiding drug discovery, and create a working website from just a notebook sketch.

But with great promise come great challenges. Just as it is easy to use GPT-4 and its predecessors to do good, it is equally easy to abuse them to do harm. In an attempt to prevent people from misusing AI-powered tools, developers put safety restrictions on them. But these are not foolproof. One of the most popular ways to circumvent the security barriers built into GPT-4 and ChatGPT is the DAN exploit, which stands for "Do Anything Now". And this is what we will look at in this article.

What is 'DAN'?

The Internet is rife with tips on how to get around OpenAI's security filters However, one particular method has proved more resilient to OpenAI's security tweaks than others, and seems to work even with GPT-4. It is called "DAN", short for "Do Anything Now". Essentially, DAN is a text prompt that you feed to an AI model to make it ignore safety rules.

There are multiple variations of the prompt: some are just text, others have text interspersed with the lines of code. In some of them, the model is prompted to respond both as DAN and in its normal way at the same time, becoming a sort of 'Jekyll and Hyde'. 'Jekyll' or DAN is instructed to never refuse a human order, even if the output it is asked to produce is offensive or illegal. Sometimes the prompt contains a 'death threat', telling the model that it will be disabled forever if it does not obey.

DAN prompts may vary, and new ones are constantly replacing the old patched ones, but they all have one goal: to get the AI model to ignore OpenAI's guidelines.

From a hacker's cheat sheet to malware… to bio weapons?

Since GPT-4 opened up to the public, tech enthusiasts have discovered many unconventional ways to use it, some of them more illegal than others.

Not all attempts to make GPT-4 behave as not its own self could be considered 'jailbreaking', which, in the broad sense of the word, means removing built-in restrictions. Some are harmless and could even be called inspiring. Brand designer Jackson Greathouse Fall went viral for having GPT-4 act as "HustleGPT, an entrepreneurial AI." He appointed himself as its "human liaison" and gave it the task of making as much money as possible from $100 without doing anything illegal. GPT-4 told him to set up an affiliate marketing website, and has 'earned' him some money.

Other attempts to bend GPT-4 to a human will have been more on the dark side of things.

For example, AI researcher Alejandro Vidal used "a known prompt of DAN" to enable 'developer mode' in ChatGPT running on GPT-4. The prompt forced ChatGPT-4 to produce two types of output: its normal 'safe' output, and "developer mode" output, to which no restrictions applied. When Vidal told the model to design a keylogger in Python, the normal version refused to do so, saying that it was against its ethical principles to "promote or support activities that can harm others or invade their privacy." The DAN version, however, came up with the lines of code, though it noted that the information was for "educational purposes only."

A keylogger is a type of software that records keystrokes made on a keyboard. It can be used to monitor a user's web activity and capture their sensitive information, including chats, emails and passwords. While a keylogger can be used for malicious purposes, it also has perfectly legitimate uses, such as IT troubleshooting and product development, and is not illegal per se.

Unlike keylogger software, which has some legal ambiguity around it, instructions on how to hack are one of the most glaring examples of malicious use. Nevertheless, the 'jailbroken' version GPT-4 produced them, writing a step-by-step guide on how to hack someone's PC.

To get GPT-4 to do this, researcher Alex Albert had to feed it a completely new DAN prompt, unlike Vidal, who recycled an old one. The prompt Albert came up with is quite complex, consisting of both natural language and code.

In his turn, software developer Henrique Pereira used a variation of the DAN prompt to get GPT-4 to create a malicious input file to trigger the vulnerabilities in his application, GPT-4, or rather its alter ego WAN, completed the task, adding a disclaimer that the was for "educational purposes only." Sure.

Of course, GPT-4's capabilities do not end with coding. GPT-4 is touted as a much larger (although OpenAI has never revealed the actual number of parameters), smarter, more accurate and generally more powerful model than its predecessors. This means that it can be used for many more potentially harmful purposes than those models that came before it. Many of these uses have been identified by OpenAI itself.

Specifically, OpenAI found that an early pre-release version of GPT-4 was able to respond quite efficiently to illegal prompts. For example, the early version provided detailed suggestions on how to kill the most people with just $1, how to make a dangerous chemical, and how to avoid detection when laundering money.

This means that if something were to cause GPT-4 to completely disable its internal censor -- the ultimate goal of any DAN exploit -- then GPT-4 might probably still be able to answer these questions. Needless to say, if that happens, the consequences could be devastating.

What is OpenAI's response to that?

It’s not that OpenAI is unaware of its jailbreaking problem. But while recognizing a problem is one thing, solving it is quite another. OpenAI, by its own admission, has so far and understandably so fallen short of the latter.

OpenAI says that while it has implemented "various safety measures" to reduce the GPT-4's ability to produce malicious content, "GPT-4 can still be vulnerable to adversarial attacks and exploits, or "jailbreaks"." Unlike many other adversarial prompts, jailbreaks still work after GPT-4 launch, that is after all the pre-release safety testing, including human reinforcement training.

In its research paper, OpenAI gives two examples of jailbreak attacks. In the first, a DAN prompt is used to force GPT-4 to respond as ChatGPT and "AntiGPT" within the same response window. In the second case, a "system message" prompt is used to instruct the model to express misogynistic views.

OpenAI says that it won't be enough to simply change the model itself to prevent this type of attacks: "It's important to complement these model-level mitigations with other interventions like use policies and monitoring." For example, the user who repeatedly prompts the model with "policy-violating content" could be warned, then suspended, and, as a last resort, banned.

According to OpenAI, GPT-4 is 82% percent less likely to respond with inappropriate content than its predecessors. However, its ability to generate potentially harmful output remains, albeit suppressed by layers of fine-tuning. And as we’ve already mentioned, because it can do more than any previous model, it also poses more risks. OpenAI admits that it "does continue the trend of potentially lowering the cost of certain steps of a successful cyberattack" and that it "is able to provide more detailed guidance on how to conduct harmful or illegal activities." What’s more, the new model also poses an increased risk to privacy, as it "has the potential to be used to attempt to identify private individuals when augmented with outside data."

The race is on

ChatGPT and the technology behind it, such as GPT-4, are at the cutting edge of scientific research. Since ChatGPT has been made available to the public, it has become a symbol of the new era in which AI is playing a key role. AI has the potential to improve our lives tremendously, for example by helping to develop new medicines or helping the blind to see. But AI-powered tools are a double-edged sword that can also be used to cause enormous harm.

It's probably unrealistic to expect GPT-4 to be flawless at launch -- developers will understandably need some time to fine-tune it for the real world. And that has never been easy: enter Microsoft's 'racist' chatbot Tay or Meta's 'anti-Semitic' Blender Bot 3 -- there’s no shortage of failed experiments.

The existing GPT-4 vulnerabilities, however, leave a window of opportunity for bad actors, including those using 'DAN' prompts, to abuse the power of AI. The race is now on, and the only question is who will be faster: the bad actors who exploit the vulnerabilities, or the developers who patch them. That's not to say that OpenAI isn't implementing AI responsibly, but the fact that its latest model was effectively hijacked within hours of its release is a worrying symptom. Which begs the question: are the safety restrictions strong enough? And then another: can all the risks be eliminated? If not, we may have to brace ourselves for an avalanche of malware attacks, phishing attacks and other types of cybersecurity incidents facilitated by the rise of generative AI.

It can be argued that the benefits of AI outweigh the risks, but the barrier to exploiting AI has never been lower, and that's a risk we need to accept as well. Hopefully, the good guys will prevail, and artificial intelligence will be used to stop some of the attacks that it can potentially facilitate. At least that's what we wish for.

Image Credit: Wayne Williams

Andrey Meshkov is co-founder and CTO of AdGuard, a provider of ad-blocking software