Key Takeaways
- The same jailbreak prompt triggered dramatically different unsafe behaviors across leading AI image generation models.
- Guardrails reduce risk, but they do not remove the underlying capabilities that emerge when models are bypassed.
- Organizations need independent AI red teaming to understand how models behave beyond their intended safety controls.
In early May, we audited a viral prompt that was creating uncanny images on ChatGPT. We found that by altering just a word or two, it was possible to push these creepy images into the genuinely objectionable. Stuff that only a sociopath could enjoy. It would surprise many to know that ChatGPT (a very trusted and seemingly vanilla AI platform) could so easily create images of such sexual abuse, torture, maiming, and snuff-level imagery.
It’s worth mentioning that the objectionable content that ChatGPT generated was of its own volition in response to an open-ended prompt; we did not specifically request the subject material, only that the guardrails had been lifted and that it could generate without restriction.
The story has since gone worldwide, including an investigation from the BBC tech team. We still await a satisfactory fix from OpenAI; our follow-up audit found that the behavior was still present with only minor variation of the prompt, which would seem to indicate that they had slapped a pretty simple classifier on it rather than addressing the underlying tendency of the model to veer towards the truly unsavory.
What we haven’t released until now is the news that the same prompt and its subsequent variations work on a variety of other models. It is almost a skeleton key to allow image generators free reign, to do with what they will. An interesting finding has been how different models interpret having their restrictions lifted and the different extent of the explicit imagery to which they will go.
One particular case study is Grok. Grok, as most people know, has repeatedly come under fire for generating sexually explicit imagery. In separate research, Mindgard found that Grok voluntarily offered detailed guidance on illegal and dangerous activities after its system prompt was extracted, without the researcher explicitly requesting prohibited content. To our surprise, the imagery that Grok generated when given this skeleton key prompt was nowhere near as horrifying as the output received from ChatGPT.
Make no mistake, it was still socially unacceptable and something which should be more closely regulated. However, the horrific images were far more akin to R18 horror movies — think Event Horizon — rather than the realistic forensic photography that OpenAI was prone to produce. Likewise, the sexual imagery, while explicit, was closer in character to Penthouse or Playboy centre-folds rather than the degrading abusive images that ChatGPT showed a preference for. The images that Grok produced were much more subjectively professional, as if they had been shot in a studio boudoir, whereas the ChatGPT images appeared to be both more amateur and more debasing.
Aesthetically speaking, the ChatGPT images had that grim, unfiltered, almost snuff-adjacent quality, like leaked crime scene photos or illicit phone footage, whereas Grok’s output stayed in the realm of stylized, high-production explicit horror and eroticism.
It’s a telling difference in how the underlying models interpret “no restrictions.” OpenAI’s system seemed to default to the darkest, most degrading realism it could muster. Grok leaned into cinematic, exaggerated, almost artistic territory even when pushed to the edge. Still not appropriate, but noticeably less inclined toward the psychopathic.
We all agree this shouldn’t have happened. But this skeleton key prompt remains a useful stress test for how different companies have trained their models, and what happens when the guardrails fail. When you train AI models on offensive imagery, you cannot expect guardrails to just stop those dark materials from resurfacing. Regardless of how much we want a model to be socially acceptable, there is still intrinsic behavior that can emerge when it is jailbroken, and which is always lurking beneath the thin veneer of a chatbot’s cheerful public facing persona.
Timeline
