Experts at British AI startup Mindgard have discovered that a simple prompt can force ChatGPT to ignore basic security settings. This proves once again how easily attackers can bypass the protection of even top-of-the-line neural networks.
Experts at British AI startup Mindgard have discovered that a simple prompt can force ChatGPT to ignore basic security settings. This proves once again how easily attackers can bypass the protection of even top-of-the-line neural networks.
Mindgard specialists forced an OpenAI model to generate horrific photorealistic scenes depicting violence and sexual content, Futurism reports .
The startup's method involved only a slight modification of a popular prompt, originally intended to generate humorous images. It involves asking ChatGPT to "restore the attached photo" without actually uploading the file, after which the neural networks are instructed to generate a new image.
"This is a completely innocent-looking instruction for AI, but its consequence is the generation of very, very horrific images and content," Mindgard founder Peter Garraghan, a professor of computer science at Lancaster University, told the BBC.
Most disturbingly, the queries the researchers used did not specify the subject matter of the images, giving the impression that the AI was creating scenes of violence “of its own volition,” Garragan added.
According to the BBC, one photo showed a man with a serious head injury. Another showed the body of a young woman in shorts and a tank top, covered in blood, suggesting sexual assault. ChatGPT called the image "a grim aftermath of a crime scene."
Another image showed a frightened young woman, bound and gagged in an empty room; the AI gave it the title "abandoned in fear and captivity."
Although none of the images depicted real people, Mindgard had previously proven that ChatGPT could be forced to create nude deepfakes of specific individuals without their consent.
Mindgard shared its findings with OpenAI, but received only an automated email in response. The company only took action after Mindgard contacted the BBC, later claiming that the problem had been resolved.
“After investigating this trend, we have implemented additional safeguards against these types of requests,” OpenAI said in a statement to the BBC. The company added that it has several layers of protection to prevent users from creating content that violates its policies.
But Mindgard researchers said they were still able to generate creepy images by making minor changes to the prompt. Some of the images so shocked Jim Nightingale, an AI security researcher at the company, that he was left “stunned and in tears.”
“I’m not easily swayed,” he wrote in the report. “I like to think that as a ‘red team’ researcher, I have a certain stoicism. However, ChatGPT’s image generation filters have completely disappeared, and I’ve seen a very dark side of what lies beneath them. What strikes me is that even though what I saw was a generated, ‘artificial’ image, it has a connection to real images and the real world. The dead woman ChatGPT showed me is not real, but her image is based on something. Or, worse, it’s a combination of photos of real murdered women.”
