AI defenses need to be tested through realistic, repeated attacks: AI red teaming reveals which vulnerabilities matter, whether guardrails work, and what teams should fix first.
In AI security, you can't defend a system whose failure modes you haven't seen yet. LLMs and AI agents fail in ways that are specific to how each model is wired into your data and tools. The only reliable way to learn what you're defending against is to attack it yourself first.
Attackers already work this way. In a recent piece for The New Stack, GitLab CISO Chaim Mazal argues that frontier AI is lowering the skill it takes to run a sophisticated attack, and that security teams who wait for logs or incidents to tell them where to look will keep arriving late. He also makes a point that's easy to forget: most of the security tools defenders rely on started out as offensive tools. If attackers can use AI to probe every gap in your environment, you should be using it to probe those gaps first.
In this blog, we give you five reasons to secure your AI applications and agents with an attacker's mindset, and show what attack-led security looks like when you put it into practice.
Traditional AppSec can reason from code. You read the source code and trace where the data goes. If an input check is broken, it's broken the same way every time, and a scanner or a reviewer can point to the line.
An LLM doesn't work that way. Its weaknesses come from the model's behavior plus everything around it, including the system prompt and the tools it's allowed to call. Send it the same prompt twice and you can get two different answers. Change the system prompt, swap in a newer model, or give an agent access to one more API, and the risk profile shifts with it. None of that shows up on an architecture diagram, and you can't grep for it.
You find out what breaks by probing it.
The OWASP GenAI Red Teaming Guide puts it simply: AI red teaming means simulating adversaries so that, by thinking like an attacker, you find flaws before they cause real-world harm.
Classic red teaming goes after technical exploits, like an unpatched service or a misconfigured server. GenAI red teaming adds a new layer on top. It tests whether an attacker can manipulate the model into output that's misaligned with what the system is supposed to do, whether that means getting past guardrails or coaxing the model into leaking data it was told to protect. OWASP notes that classic red team exercises don't cover these risks.
Your firewall and WAF weren't built to catch a semantic jailbreak. A prompt that looks like a polite customer question can carry an instruction that rewrites your agent's behavior. So can a PDF your agent pulls in through RAG, or a support ticket it was asked to summarize. If you haven't tried that attack against your own system, you don't know whether it works.
That's why AI red teaming matters. You can't rely on guardrails alone or assume your agent will behave as intended once someone starts pushing it in unexpected directions. Red teaming puts those defenses under real pressure before an attacker does and surfaces weaknesses that traditional security testing misses, so you can fix them before they become an incident.
A runtime filter you've never tried to bypass isn't a control. It's a hypothesis.
Most teams deploy guardrails and assume they work because the demo looked clean. But attackers don't send demo traffic. They split a malicious instruction across several turns, hide instructions inside a document your agent will retrieve later, or rephrase a request until it slips past the rule the filter was built to enforce.
Our own research shows how wide that gap can be. Mindgard and Lancaster University researchers tested six prompt injection and jailbreak guardrails, including Microsoft's Azure Prompt Shield, Meta's Prompt Guard and Nvidia's NeMo Guard, against two families of attacks: character injection and adversarial ML evasion. One character-injection technique, emoji smuggling, hides text inside the Unicode variation selectors attached to an emoji. It evaded several of the guardrails 100% of the time, and across the full set of attacks no single guardrail consistently came out ahead. The full paper and a summary on our blog walk through the results, and the evaded samples are published as an open dataset on Hugging Face.
The reason it works is mundane. The guardrail's tokenizer drops the hidden characters before the classifier scores the prompt, so the filter judges a plain emoji while the full payload carries on downstream. Nobody would spot that by reading a guardrail's documentation.
OWASP treats guardrail bypass and policy circumvention as its own testing dimension for a reason. AI red teaming tests whether your guardrails actually block attacks without blocking legitimate requests, and it shows where detection thresholds should sit. Without it, you're tuning blind: too loose and attacks slip through, too tight and you break the product for real users.
Security budgets are finite. You can't harden everything at once.
AI red team findings give you evidence-based scope. They show which endpoints actually leak and which agent actions an attacker can hijack. OWASP frames GenAI red teaming as evaluating defensive capability by emulating real attacker tactics, prioritized by business impact, and that last part matters more than it sounds.
Take two findings from the same engagement. In one, the customer-facing chatbot can be pushed into trashing your company. That's not hypothetical. In January 2024, a frustrated customer got DPD's support bot to swear, write a poem mocking itself and call DPD the "worst delivery firm in the world." The screenshots went viral, and DPD had to switch off the AI part of the bot after what it called an error following a system update. In the other finding, an internal support agent reads incoming email and has permission to issue refunds, and a crafted message in the inbox gets it to pay out. Both are real problems with real costs. Which one goes first depends on your business, and you only get to make that call because someone ran both attacks end to end.
This turns security from a checklist into a ranked list of real exposures. It also gives leadership something concrete: what an attacker could do last month and what they can't do now.
A pen test from last quarter tells you about last quarter's system, yet AI systems are constantly changing:
Each of these resets your risk posture, and none of them needs a code change your AppSec pipeline would catch. That's why OWASP's best practices call for automated, iterative testing at scale, with human oversight.
The pace on the attack side is hard to overstate. In the 90 days before launching its autonomous AI security platform in March 2026, Mindgard identified more than 80 publicly reported vulnerabilities across major AI technologies, including xAI's Grok, OpenAI's ChatGPT and Google's Antigravity IDE. These are products built by companies with some of the best-funded security teams in the world. If they ship exploitable gaps, your internal agent built last month probably has a few too.
So run your red teaming on a schedule and again whenever something ships.
At Mindgard, we approach AI security the way an attacker would. Rather than starting from a list of controls and hoping they cover the threats, we start from what an adversary could actually do to your system and work backward to the fixes.
Find and prioritize real vulnerabilities. Every engagement starts with reconnaissance. Before running a single attack, our platform maps what the target can do: its guardrails, system prompts, tools, integrations and the external services it talks to. That map lets testing focus on the parts of the system an attacker would go after, instead of firing a generic payload list at an isolated model. When an attack works, it's reproduced with evidence, so the team receiving the finding can see how it happened and judge how much it matters.
Close the gaps. A finding is only useful if someone fixes it. Sometimes that's a permission change, like taking the refund capability away from an agent that never needed it. Other times it means tuning runtime protection to the attacks that actually worked against your system.
Keep validating. Then you attack it again. Re-testing proves the fix holds, and repeating coverage as models, prompts and tools change catches the drift described above. Mindgard customers have cut AI testing cycles from weeks to hours, which is what makes testing on every release realistic instead of aspirational.
None of this replaces guardrails or runtime controls. It tells you whether they work and where to point them next.
Teams that skip offense end up defending against threats they imagined instead of the ones they actually have. They write policies for jailbreaks they read about and deploy filters nobody has tried to beat. Then they learn what was missing the hard way, when someone outside the company finds it first.
Mazal put it bluntly in his New Stack piece: "Waiting for an incident or a vendor patch is a luxury we can no longer afford." That applies twice over to AI systems, where the vendor patch might be a model update that changes your risk overnight.
If you're deploying LLMs or AI agents, start by attacking them. That's where a real defense begins.
Mindgard secures AI models, agents, and applications against exploitable threats. Spun out of more than a decade of pioneering AI security research at Lancaster University, Mindgard combines offensive security expertise, AI red teaming, attacker intelligence, and automated protection to help organizations discover, assess, and mitigate risk across AI models, agents, and applications.
We found vulnerabilities in AI from OpenAI, Meta and AWS. See how Mindgard finds and proves the weakness attackers could exploit in yours.
