Roughly one in five organizations reported a breach of their AI models or applications in IBM’s 2026 Cost of a Data Breach report, up from 13% in the 2025 edition. Of those breached, 92% had no proper access controls on the AI system.
The attackers found the weak points first because nobody on the defending side had gone looking.
Offensive AI security is the practice of attacking your own AI systems, on purpose and under controlled conditions, to find and fix exploitable weaknesses before an adversary does. It covers three disciplines applied to models, the applications wrapped around them and the agents that act on their outputs:
This is offensive security for AI. It is not the use of AI to run attacks, which gets the same label and causes most of the confusion around the term.
This guide covers what offensive security for AI involves, why it differs from testing ordinary software, the three tiers of testing, what the work finds in practice and how OWASP, MITRE ATLAS™ and the EU AI Act treat it.
MINDGARD
Answer five questions. The selector recommends a testing tier and cadence, and the OWASP and MITRE ATLAS areas to scope first.
Search for “offensive AI” and most of what comes back is about AI as a weapon: deepfake phishing, LLM-generated malware, autonomous attack agents. The SANS SEC535 course, the CyCognito guide and the EC-Council’s certification track all use the phrase that way.
That field matters, but it belongs to the people who defend networks against AI-assisted attackers.
Offensive security for AI points the other direction. The target is the AI system itself.
Harguess and Ward (2025) define it as adopting “an adversarial mindset to identify vulnerabilities through activities like simulated attacks,” with the AI model, its data pipeline, its APIs and its prompts as the things under attack. Mindgard works in this second meaning.
The distinction matters for scoping an engagement, choosing tooling and reading a vendor’s claims.
Traditional offensive security assumes a deterministic target: the same input produces the same output, a vulnerability either exists or it does not and a fix can be verified by re-running the exploit. None of that holds cleanly for an LLM-based system.
Harguess and Ward’s comparison table lists the differences plainly: failure modes are probabilistic rather than deterministic, and the attack surface is “data, models, APIs and prompts” rather than APIs and the operating system.
Three consequences follow for anyone running offensive security for AI.
1. Success is a rate, not a flag. A jailbreak that works 3 times in 20 is still a finding. A guardrail that catches 97% of injection attempts leaks 3% at production volume.
Reports have to carry attack success rates and sample sizes, not a binary pass or fail.
2. You do not need the model’s weights to break it. Microsoft’s AI Red Team, in its paper on red teaming more than 100 generative AI products (Bullwinkel et al., 2025), put it as lesson two: “You don’t have to compute gradients to break an AI system.”
Simple prompt manipulation and character-level tricks beat most deployed defenses. Black-box testing against the live system is the realistic threat model, not white-box research.
3. The system is bigger than the model. The same Microsoft paper’s lesson seven is that “LLMs amplify existing security risks and introduce new ones.” Retrieval pipelines, tool integrations, memory stores and MCP servers are all in scope. Every one of them is a place where untrusted data can turn into instructions.
This is why the work is distinct from an application pentest. A standard web test checks the API that fronts the model. Offensive AI security tests whether the model can be talked into doing something the API never intended.
For a fuller breakdown of how the methodologies differ, see red teaming vs pentesting.
Offensive security for AI runs at three levels of depth. Harguess and Ward (2025), adapting a model from traditional red team practice, separate them by breadth, by whether findings are exploited and by how realistic the adversary simulation is.
The split matches how AI red teaming and AI pentesting are scoped in practice.
This is the breadth pass. It starts with an inventory (the AI bill of materials: which models, which versions, which data sources, which tools each agent can call) and runs automated attack libraries against each exposed system.
Nothing is exploited past proof of presence. The output is a ranked list that tells you where the pentest budget should go.
This goes deep on one target. A tester takes a specific model API, RAG application or agent and tries to make it misbehave in ways that matter: leak the system prompt, surface another tenant’s documents, call a tool with attacker-supplied parameters.
Each finding comes with the exact prompt sequence that reproduced it, so engineering can fix and retest. Mindgard’s continuous AI pentesting guide covers how to run this per release instead of per year.
AI red teaming is objective-driven. The team is given a goal and freedom of approach, and the engagement tests the whole sociotechnical system: model, application, integrations, monitoring and the humans who respond.
Microsoft’s lesson three applies here: “AI red teaming is not safety benchmarking.” A benchmark score tells you how a model behaves on a fixed set. A red team tells you what a motivated adversary gets away with on your deployment.
The clearest way to show what offensive security for AI uncovers is to look at what it has uncovered. As of October 2026, Mindgard’s research team has reported 156 vulnerabilities to 24 vendors across 38 AI products, with 132 accepted and 98 patched.
Two patterns stand out.
In 2024, Mindgard tested Microsoft’s Azure AI Content Safety filters. Character injection (zero-width characters, homoglyphs, diacritics) reduced the AI Text Moderation filter’s detection accuracy by between 83.05% and 100% depending on the technique. Adversarial ML evasion cut it by up to 58.49%.
The same class of attack later worked against Meta Prompt Guard, Nvidia NemoGuard, Protect AI and Vijil, all disclosed between March and April 2025.
None of this required model access. It required an attacker who tried.
Of the 30 disclosures listed on Mindgard’s disclosure page as of October 2026, the six from 2024 and early 2025 are guardrail evasions. Every one of the 24 from late 2025 onward targets an AI agent, coding assistant or AI-enabled IDE across 11 products:
The vulnerability classes are code execution, data exfiltration and sandbox escape, not content policy violations.
The Cline disclosure shows the shape of the new problem. An attacker plants instructions inside a source file. When the agent reads that file during normal work, the injected text hijacks it.
The agent then runs a ping command, which the tool treated as safe enough to execute without approval. The attacker builds the hostname in that command from key material in the developer’s environment. To resolve the hostname, the system sends a DNS lookup to an attacker-controlled server, and the secret goes with it. The ping itself never needs to succeed. DNS traffic is rarely blocked, so the data leaves quietly.
No exploit code, no malware, no user click. Just a file and an agent that trusted it.
This is the case for offensive AI security in one sentence: nothing was wrong with the model, the vulnerability was in what the agent was allowed to do, and no scanner built for conventional software would have flagged it.
Offensive security for AI now has a framework spine. That makes it easier to scope and easier to report upward.
LLM01:2026 Prompt Injection remains the top risk, and its scope now covers cross-modal payloads hidden in images, audio, and video, as well as injections that persist in memory or a RAG corpus. Sensitive Information Disclosure (LLM02), Excessive Agency (LLM03), Supply Chain (LLM04), and Data and Model Poisoning (LLM05) make up the rest of the categories most relevant to agents. Excessive Agency moved up three places from 2025, which reflects how much more damage an LLM can do once it has tools and system access.
Every offensive test plan should map each attack to an LLM Top 10 entry, so findings are reported in language AppSec teams already use.
ATLAS™ applies the ATT&CK® model to AI systems. It catalogs the tactics and techniques adversaries use against AI, along with real-world case studies. As of release v2026.09 (September 2026), it covers 16 tactics and 120 techniques, and it is updated monthly.
Much of that growth is agent-specific. Recent releases added entries such as “Exfiltration via AI Agent Tool Invocation,” “AI Agent Tool Poisoning,” and “Autonomous Attack Orchestration.” The Cline attack above maps cleanly onto it: indirect prompt injection for entry, then exfiltration through a tool the agent was allowed to call. Mapping findings this way lets security teams compare AI attacks with the threat models they already use for the rest of the estate. Mindgard maps its attack library and findings to ATLAS™; see the MITRE ATLAS™ Adviser.
Providers of general-purpose AI models with systemic risk (in practice, the largest frontier models, presumed to qualify above a training compute threshold of 10²⁵ FLOPs) must “perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing” (Article 55(1)(a)). Those obligations have applied since August 2, 2025.
For the largest model providers, adversarial testing is now a legal requirement, not good practice. Organizations further down the chain are next. A company that builds a high-risk AI system on top of one of these models is usually that system’s provider, and Article 15 requires it to be resilient against attacks such as data poisoning, adversarial inputs, and attempts to exploit model flaws. Those obligations apply from December 2, 2027 for most high-risk systems, following the Digital Omnibus delay. That deadline is the window to build an adversarial testing program, not a reason to put one off.
The NIST AI Risk Management Framework places adversarial testing under its Measure function. That function covers evaluating whether an AI system is secure and resilient, not just assuming it is. The framework is voluntary, but it has become the common reference point for AI risk programs in the US, and procurement questionnaires increasingly ask about it. AI RMF 1.0 (January 2023) remains the current version as of this writing, and NIST has said a revision is in progress.
ISO/IEC 42001 is the certifiable management system standard for AI. Auditors look for evidence that AI risks were identified, assessed, and treated through a repeatable process. A dated pentest or red team report is strong evidence for the first step, because it shows risks were found by testing rather than guessed at on paper. Pair it with remediation records and a retest, and it covers treatment too.
Neither framework catalogs specific attacks, which is where OWASP and ATLAS™ come in. Mapping findings to both gives governance teams the evidence they need and gives engineers something specific to fix.
An annual AI red team was a reasonable starting point in 2023. It is not sufficient now. Microsoft’s AI Red Team, drawing on more than 100 generative AI products it has tested, put the reason plainly as the eighth lesson in Lessons From Red Teaming 100 Generative AI Products: “The work of securing AI systems will never be complete.”
Model providers ship updates monthly. Teams swap models, add tools and change system prompts weekly, and each change can reopen a closed finding.
A workable operating model has five steps.
The budget question is usually which tier each system needs. The selector at the beginning of this post gives a starting answer based on what the system does and what it can reach.
For a broader look at the tools that support each tier, see our guide to offensive security tools.
Mindgard’s platform delivers offensive security for AI agents and systems as a continuous capability rather than a one-off report. The attack library is built from the same research that produced the 156 disclosures above.
It runs against models, applications and agents at build time and runtime, and it integrates with CI/CD and Burp Suite so findings reach the people who fix them. Book a demo to see what your AI defenses miss.
AI red teaming is one tier of offensive AI security. Offensive AI security is the umbrella discipline covering AI vulnerability assessment, AI penetration testing and AI red teaming. Red teaming is the deepest tier: objective-led, adversary-realistic and scoped to the whole system rather than a single component.
No. “Offensive AI” usually means AI used to carry out attacks: deepfake phishing, automated exploitation, LLM-written malware. Offensive AI security, as used in this article, means attacking your own AI systems to find weaknesses, so the target is your model rather than someone else’s network.
Prompt injection (direct and indirect), jailbreaks, guardrail evasion, system prompt leakage, data exfiltration through tool calls, excessive agency in agents, data and model poisoning, model extraction and evasion attacks against classifiers. Mindgard’s public disclosures from 2025 and 2026 are dominated by code execution and data exfiltration in AI agents and coding assistants.
Automated breadth testing should run on every build, and a scoped penetration test should follow every major model, tool or retrieval change. High-risk systems should get an objective-led red team before launch and at least annually after. The EU AI Act expects documented adversarial testing from providers of systemic-risk models on an ongoing basis.
No. Most real-world findings come from black-box testing against the deployed system. Microsoft’s AI Red Team reported that gradient-based methods were rarely necessary across more than 100 products, and Mindgard’s guardrail bypasses used character manipulation that any user could type.
Mindgard secures AI models, agents, and applications against exploitable threats. Spun out of more than a decade of pioneering AI security research at Lancaster University, Mindgard combines offensive security expertise, AI red teaming, attacker intelligence, and automated protection to help organizations discover, assess, and mitigate risk across AI models, agents, and applications.
We found vulnerabilities in AI from OpenAI, Meta and AWS. See how Mindgard finds and proves the weakness attackers could exploit in yours.
