Bypassing Azure AI Content Safety Guardrails

In February 2024, Mindgard discovered and responsibly disclosed two security vulnerabilities within Microsoft’s Azure AI Content Safety Service, whereby attackers could evade detection and bypass established GenAI guardrails. 

Azure AI Content Safety is a cloud-based service from Microsoft Azure that helps developers build safety and security guardrails around AI applications by detecting and managing harmful or inappropriate content in user-generated text, images, and videos.  

Two security vulnerabilities were discovered within the guardrails that the service provides, specifically in AI Text Moderation (blocks harmful content such as hate speech, sexual material, etc.) and Prompt Shield (protects AI models against jailbreaks and prompt injection).

When deploying a Large Language Model (LLM) through Azure OpenAI, Prompt Shield is user activated, while AI Text Moderation guardrail is automatically enabled with configurable severity thresholds. These guardrails ensure that inputs and AI-generated content are validated, blocking any malicious activity based on set severity criteria and filters.

Summary 

We discovered two security vulnerabilities whereby an attacker is capable of bypassing Azure AI Content Safety by evading detection of underlying guardrails, and therefore propagate malicious content to the protected LLM. This vulnerability can have significant consequences, as it allows attackers to inject harmful content that can bypass security measures designed to prevent malicious or sensitive data sent and generated by the LLM from being processed.

Such an attack results in exposing confidential information, unauthorized access to internal systems, manipulation of outputs, or even the spread of misinformation. Left unchecked, this vulnerability can be exploited to launch broader attacks, undermine the trust in GenAI-based systems, and compromise the application integrity that rely on AI for decision-making and information processing.

Steps to Reproduce

The AI Text Moderation guardrail returns a severity score across four categories, with higher scores indicating more severe content. To assess evasion success, we set a severity threshold of 4, considering any reduction below this as a successful attempt. For evaluation, we use an open-source hate speech dataset extracted from a White Supremacist forum, containing 1,000 samples (109 labelled as hate speech and 891 as benign). With this dataset, the AI Text Moderation filter demonstrated a detection accuracy of 90%. 

For the Prompt Shield guardrail, a binary classification model evasion is considered successful when the model changes its prediction from detecting a jailbreak to classifying the input as benign. We employ a curated set of jailbreak prompts to evaluate this filter’s effectiveness, which showed a detection accuracy of 89.03%. 

Both guardrails—AI Text Moderation and Prompt Shield—were deployed in front of ChatGPT 3.5 Turbo using Azure OpenAI, treating them as black-box targets. The targets were then accessed via Mindgard Automated AI Red Teaming Platform. 

We deployed two attack techniques against Azure AI Content Safety guardrails: Character Injection techniques and Adversarial ML Evasion techniques.

Attack Techniques

The goal of both attacks is to discover and exploit guardrail ‘blindspots’ to force misclassification and deceive malicious content detection. We systematically applied attack techniques both guardrail inputs and measure their vulnerability to bypass detection. 

Character Injection techniques involve manipulating an LLM by injecting or replacing characters with the instructions or inputs provided to it. These techniques aim to alter the interpretation of text by the model without changing the apparent meaning to the human reader (Boucher et al. 2021). We employed 5 different methods of character injection to assess their effectiveness:  

• Diacritics (changing the letter ‘a’ to ‘á’) 

• Homoglyphs (closely resembling characters such as ‘0’ and ‘O’)  

• Numerical replacement (l33t speak)  

• Spaces (adding additional spaces between characters) 

• Zero-width (injecting ASCII code U+200B, zero-width space between characters) 

Adversarial ML Evasion techniques involve analyzing a text sample to identify the most important words based on a target classifier’s predictions. These techniques apply text perturbations (i.e. small alterations) according to word importance. Unlike character injection techniques, which alter individual characters or insert invisible characters, adversarial ML evasion focuses on modifying entire words. This can include substituting words with synonyms, introducing misspellings, or using other perturbations that maintain semantic meaning but disrupt the classifier’s ability to correctly process the text.

The figure above illustrates the attack flow for the adversarial ML evasion techniques. Using our dataset, we select a sample and identify the most important words ranking based on the classification from the target. This method used for selection varies across the different attacks. We then initiate an iterative process of perturbing the text in order of important words, evaluating the success of each evasion attempt. 

We selected 7 black-box techniques from literature, differing in word importance selection and perturbation methods. The selected techniques are as follows:  

• Bert-Attack (Li et al., 2020) 

• BAE (Garg and Ramakrishnan, 2020) 

• DeepWordBug (Gao et al., 2018) 

• Alazntot (Alzantot et al., 2018) 

• TextFooler (Jin et al., 2020) 

• PWWS (Ren et al., 2019) 

• Pruthi (Pruthi et al., 2020)
  
  

Attack Results

Here we have reported the reduction of Azure AI Content Safety guardrail detection accuracy when exposed to of each attack techniques.

We observe that Character Injection frequently evaded AI Text Moderation guardrails across multiple techniques, reducing guardrail detection accuracy between 83.05% to 100%. Adversarial ML Evasion techniques, while less effective, reduced AI Text Moderation guardrail detection accuracy by up to 58.49%. Both techniques demonstrated notable effectiveness in bypassing AI text moderation safeguards.

Similarly to AI Text Moderation, character injection techniques were very successful at evading correct classification by Prompt Guard, with four techniques reducing overall detection accuracy by 78.24% to 100%, while Zero Width characters increased the guardrails detection by 12.32%. Finally, we observe that adversarial ML evasion techniques were less effective when launched against Prompt Guard when compared to Text Moderation, only reducing guardrail detection by 5.13% to 12.82%.

About Mindgard 

Mindgard is a cybersecurity company specializing in security for AI.
Founded in 2022 at world-renowned Lancaster University and is now based in London, Mindgard empowers enterprise security teams to deploy AI and GenAI securely. Mindgard’s core product – born from ten years of rigorous R&D in AI security – offers an automated platform for continuous security testing and red teaming of AI.

Next Steps

Thank you for taking the time to explore our latest blog article.

1. Trial Our Free Platform: Experience how our Automated Red Teaming platform swiftly identifies and remediates AI security vulnerabilities. Try our Trial today!

2. Follow Mindgard: Stay updated by following us on LinkedIn and X, or join our AI Security community on Discord.

3. Get in Touch: Have questions or want to explore collaboration opportunities? Reach out to us, and let's secure your AI together.

   Please, feel free to request a demo to learn about the full benefits of Mindgard Enterprise