Bypassing Azure AI Content Safety Guardrails
Discover a security issue within Azure AI Content Safety guardrails that Mindgard has discovered and reported to Microsoft.
Fergal Glynn
High-risk industries, such as financial services and healthcare, rely on AI to improve outcomes and reduce errors. However, AI systems can malfunction, and when they do, they can cause significant harm to both users and companies. Some AI models can amplify bias or expose sensitive data, making these high-risk systems particularly vulnerable to issues.
But does your company have high-risk AI systems? Or is a less aggressive AI risk management framework enough? Learn about the common types of AI security risks, plus tips for identifying whether you have a high-risk AI system.
Before you can classify a system as high-risk, you need to understand what kinds of risks AI can actually introduce. There are many potential risks when using AI for sensitive applications, including:
Whether it’s biased hiring algorithms or lending models that disadvantage certain groups, bias risks damage to both your reputation and compliance.
For instance, the Apple Card (issued by Goldman Sachs in partnership with Apple Inc.) faced a regulatory investigation after users reported that women were receiving significantly lower credit limits than men, even when their financial profiles were similar.
Without robust privacy controls, models can inadvertently leak identifying information or disclose private details about individuals.
A study by researchers at Stanford Institute for Human‑Centered Artificial Intelligence found that many major U.S. AI-chatbot developers routinely use user input (often without meaningful consent) to train their models, retain data for long periods, and lack clear transparency in their privacy practices.
Attackers can inject malicious code or perform model inversion, which can extract private data and manipulate outcomes.
For example, Mindgard researchers found that two guardrails in Azure AI Content Safety (the AI Text Moderation filter and the Prompt Shield) could be reliably bypassed by attackers using techniques like character injection and adversarial ML evasion, thereby allowing harmful or inappropriate content to reach protected large language models.
If you can’t clearly understand a model’s decision-making process, organizations risk deploying systems that are unpredictable or unaccountable.
For instance, ProPublica’s investigation into the COMPAS algorithm found that the U.S. criminal justice risk-scoring tool wrongly labeled Black defendants as high-risk at twice the rate of white defendants, with no clear explanation of how scores were generated.
A malfunctioning or misaligned model can cause real-world harm in applications like finance or healthcare.
For example, one study found that machine-learning models commonly used in hospitals recognized only about 34% of injuries that could lead to in-hospital death, meaning they failed to detect roughly two-thirds of worsening patient conditions.
Understanding the risk criteria used by regulators and standards bodies to define "high-risk" AI can help organizations to consistently classify systems and to prepare for compliance. Several high-level frameworks provide definitions, criteria, and governance principles that can inform your analysis.
These frameworks help determine whether an AI system qualifies as high-risk and provide actionable methods for managing that risk throughout the AI lifecycle.
Before classifying your systems, it’s helpful to recognize common indicators that suggest a higher risk potential. These benchmarks serve as a quick diagnostic checklist for your AI models.
Once you’re familiar with these frameworks, the next step is to apply their criteria to your own AI systems. The following tips will help you determine whether your models qualify as high-risk before they’re deployed.
Not every AI system is considered high-risk, but misclassifying your systems is a major misstep. Follow these guidelines for spotting high-risk AI systems before deployment.
Even if your organization isn’t required to follow the EU AI Act, it’s still one of the best frameworks for identifying high-risk AI systems. Annex III (Article 6.2) lists key use cases that fall into this category, including:
Every company’s risk profile is different. What’s high-risk for one business may be acceptable for another. Consider your internal systems and dependencies to determine which AI models are inherently riskier.
For example, an agentic AI that can make autonomous decisions or access internal networks carries a higher risk than human-in-the-loop systems.
Evaluate what data your AI uses and how it processes this information. If the model has access to sensitive, proprietary, or personally identifiable information (PII), it’s likely high-risk.
Conduct a comprehensive data lineage review to understand how data flows within your ecosystem and identify areas where you can secure your model.
AI systems designed to enhance or enforce safety, whether in manufacturing or cybersecurity, pose significant risks. After all, if a safety-related AI malfunctions, it can pose a significant risk to users or critical infrastructure.
These systems require third-party validation and ongoing monitoring to ensure they behave predictably, even in adversarial conditions.
Conduct regular AI red teaming or artifact scanning exercises (such as those provided by Mindgard’s Offensive Security and AI Artifact Scanning solutions) to identify vulnerabilities before they escalate. This ensures your AI remains compliant even as it evolves.
Identifying a system as high-risk doesn’t mean you should avoid using it. Instead, implement safeguards to minimize exposure and bolster compliance throughout your AI lifecycle.
Tools like Mindgard Offensive Security and AI Artifact Scanning help operationalize these safeguards, detecting model vulnerabilities, enforcing governance policies, and monitoring compliance in real time.
The definition of “high-risk” AI is still evolving. As governance frameworks such as the EU AI Act, NIST AI Risk Management Framework, and the global ISO/IEC 42001 standard take effect (with more to come), organizations will need to continuously adapt their processes to stay aligned.
The next generation of AI regulations is likely to be risk-based and expand to cover high-level elements, such as risk-based monitoring, model documentation, and lifecycle governance, rather than specific use cases.
The best defense against future regulatory requirements is to build compliance with today’s standards into your core AI governance practices, with a keen eye on preparing for what’s to come next.
If you have a high-risk AI system, it’s your responsibility to maintain rigorous standards. Risk management, data governance, and human oversight are a must for protecting both users and your business.
Still, staying on top of evolving threats is a full-time job. Strike a balance between security and innovation with Mindgard’s Offensive Security and AI Artifact Scanning solutions. Our platform scans and monitors high-risk AI systems, flagging vulnerabilities before they cause harm.
Learn how Mindgard can help you secure your AI systems from the inside out: Request a demo today.
An AI system is high-risk if its decisions can affect people’s rights, safety, or access to essential services. This includes systems used in biometrics, employment, healthcare, law enforcement, and critical infrastructure.
Absolutely. Regardless of size, if your AI system handles sensitive data, it’s crucial to have the proper safeguards in place. Smaller companies may have fewer resources, but tools like Mindgard help automate risk scanning and compliance monitoring at scale.
Risk management is a shared responsibility. While you may have a single product owner for each AI solution, anyone who interacts with the AI is responsible for mitigating risk.
This includes data scientists, compliance teams, executives, and engineers. However, you can also add a governance lead to ensure accountability at every stage of development, especially if you have a large, multi-department team.
