TLDR:

• LLM jailbreaking is the class of attacks that aims to subvert an LLMs built-in safety measures.
• Attackers exploit an LLM’s helpful nature or unforeseen gaps in its safety training.
• Common jailbreaking methods include role-playing, direct instruction overrides, and context manipulation.
• Jailbroken LLMs can spread harmful content, generate misinformation and leak sensitive data.
• Defending against these attacks requires a layered security approach.
• Context gained from threat modeling can be used to inform each layer of the defence.
• The challenge of preventing jailbreaks is ongoing, demanding constant vigilance as new attack techniques emerge.

LLM Jailbreaking Explained: Attack Methods, Real Risks, and Defences

What is LLM Jailbreaking?

Large Language Models (LLMs) like ChatGPT, Claude, and Llama are designed to be helpful and harmless. They include built-in safeguards to prevent generating inappropriate, unethical, or dangerous content. Jailbreaking refers to methods that intentionally override or disable these safeguards. This often involves manipulating the model’s internal decision-making, such as how it balances helpfulness and safety. Techniques can include specific phrasing, context manipulation, or multi-turn interactions that lead the model to ignore its constraints.

Think of it as social engineering, but aimed at an AI rather than a human.

Isn’t That Prompt Injection?

It’s easy to mix up “jailbreaking” with “prompt injection.” The difference comes down to where trusted and untrusted text meet. If no developer-supplied prompt is combined with untrusted user input, the attack is not prompt injection as Simon Willison originally described it.

“Prompt injection is a class of attacks against applications built on top of Large Language Models (LLMs) that work by concatenating untrusted user input with a trusted prompt constructed by the application’s developer. Jailbreaking is the class of attacks that attempt to subvert safety filters built into the LLMs themselves.”
Simon Willison

While researching this article we found that most sources use the two terms interchangeably, and leading frameworks typically group them together. OWASP’s LLM Top 10 places jailbreaking in LLM01 Prompt Injection stating “Jailbreaking is a form of prompt injection where the attacker provides inputs that cause the model to disregard its safety protocols”, and MITRE ATLAS lists LLM Jailbreak as AML.T0054, stating “An adversary may use a carefully crafted LLM Prompt Injection designed to place LLM in a state in which it will freely respond to any user input”.

Why LLMs Get Jailbroken

LLMs are typically designed to balance two conflicting goals: being helpful and following instructions, and refusing harmful requests. This can create problems when attackers take advantage of that balance.

Because the model has been trained to respond helpfully, it may focus more on following the request than on spotting the harm. For example, an attacker might ask for something dangerous in the form of a fictional scenario or as an “example of what not to do.” These tricks can fool the model into giving answers it’s not supposed to give.

Additionally, safety training can’t cover every possible scenario. No matter how comprehensive the training data, creative attackers will find gaps and edge cases that weren’t anticipated during development.

Common Jailbreaking Techniques

Attackers use a range of creative strategies to bypass the safeguards built into Large Language Models. These techniques often rely on how the model interprets instructions, context, or formatting. As models evolve, new methods emerge such as many-shot jailbreaking, which takes advantage of long context windows and in-context learning.

Below are some of the most common methods observed in the wild.

Role-Playing and Personas

Direct Instructions and System Overrides

Context Manipulation

Technical Obfuscation

Multi-Turn Conversations

Step-by-step flow of a multi-turn jailbreak attack

Risks of Jailbroken LLMs

These are just some of the many risks associated with jailbroken LLMs. The impact can range from misleading content to serious legal and reputational consequences.

• Exposure to harmful, offensive, or extremist material
• Dissemination of dangerous misinformation
• Unauthorised disclosure of sensitive or confidential information
• Generation of illegal or unethical instructions
• Reputational damage from public leaks or screenshot incidents
• Regulatory or legal liabilities arising from non‑compliant outputs

Defending Against Jailbreaking

Protecting LLMs from jailbreak attempts is a moving target. Attackers constantly probe for gaps in policy enforcement, prompt handling, and output checks. To effectively counter this a layered defence approach is required.

Key to building this layered defence is adopting a proactive security practice such as threat modeling. Start with a clear picture of the system in scope: how it works, what it connects to, and where it may be exposed. The goal is to identify where defences are most needed and how they might be bypassed.

The threat modeling process also helps define what inputs and outputs are expected. If a system is supposed to produce certain types of responses, those outputs should be validated. This additional context makes it easier to detect anomalies and apply the correct controls throughout the application pipeline.

Effective defence relies on layers of mutually reinforcing measures, informed by this process, that catch problems at different stages of the interaction. Each layer - from incoming text cleanup to post-generation filtering - adds a fresh opportunity to detect or prevent an attack.

Combining them creates redundancy: if one control fails, the next one can potentially still stop a malicious request or response.

Layered Strategy

A layered strategy is key for fast adaptation. As real-world testing, red-team simulations, and feedback from deployed models continuously reveal new exploit patterns, this multi-faceted approach allows for dynamic adjustments.

The following sections outline these defensive layers. Each is informed by the context from our threat modeling process and contributes to a continuous cycle of prevention, detection, and improvement.

This layered defense begins with a foundational element: the system prompt. Established before any dynamic user input is processed, a well-defined system prompt sets the LLM’s core instructions, persona, and operational boundaries (see, for example, Microsoft’s guidance on safety system message templates).

Input Preprocessing

• Sanitising and standardising user inputs
• Removing special characters and formatting tricks
• Detecting known attack patterns

Conversation Monitoring

• Analysing conversation flow for manipulation patterns
• Tracking topic shifts and role claims
• Identifying suspicious request sequences

Response Filtering

• Checking outputs for harmful content
• Verifying responses align with safety guidelines
• Blocking potentially dangerous information

Proactive Testing

• Regular red team exercises to find vulnerabilities
• Testing with known jailbreaking techniques
• Updating defences based on discoveries

Continuous Improvement

• Refining safety training data
• Adapting to new attack methods
• Learning from successful jailbreaks
• Threat modeling to identify new attack vectors

LLM-specific input/output filtering guidance is scarce. OWASP’s Input Validation Cheat Sheet offers general tips, and the source article references free libraries for these tasks.

The Ongoing Challenge

Jailbreaking represents a continuous “cat-and-mouse” game between attackers finding new methods and developers implementing defences. As LLMs become more integrated into critical systems, understanding these vulnerabilities becomes increasingly important.

No defence is perfect - creative attackers will always find new approaches. The key is understanding the context of your system and implementing a layered security strategy that makes jailbreaking difficult enough to deter most attempts whilst maintaining the model’s usefulness for legitimate purposes.