What is Data Poisoning? The Silent Threat Already Inside Your AI Pipeline

Data poisoning attacks are some of the most insidious and difficult to detect attacks against AI models. Poisoned training or retrieval data may never exhibit obvious failures until well into the lifespan of your model. Data poisoning is #4 on OWASP'S Top 10 for LLM Applications (2025). This attack purposefully alters your data, resulting in unexpected behaviors from your AI. Data poisoning can be as innocent as degraded performance, but can also lead to severe security vulnerabilities.

The risk landscape is expanding rapidly. JFrog's 2025 Software Supply Chain Report discovered malicious models uploaded to Hugging Face increased by 6.5x in just one year. Hugging Face is one of the leading repositories enterprises rely on for public pretrained weights and shared checkpoints. IBM's 2025 Cost of a Data Breach Report reveals 13% of organizations suffered a breach of AI models or applications in 2024, and found that 97% of those affected didn't have basic AI access controls enabled. Those controls are available. They're just not widely implemented yet.

Intended for AI security practitioners and ML engineers who build and maintain LLM systems, this guide walks through mechanics of poisoning attacks, injection points in the pipeline, methods for detection, and controls most effective for prevention.

How Data Poisoning Affects AI and LLMs

Training machine learning models, especially at the scale required for useful AI, needs large amounts of data. So large in fact that it's impossible for teams to verify every input themselves. As such, teams have to trust their data pipelines to an extent. Bad actors are taking advantage of that trust.

“There are many opportunities for bad actors to corrupt this data — both during an AI system’s training period and afterward, while the AI continues to refine its behaviors by interacting with the physical world. This can cause the AI to perform in an undesirable manner. Chatbots, for example, might learn to respond with abusive or racist language when their guardrails get circumvented by carefully crafted malicious prompts.”

- National Institute of Standards and Technology (NIST)

Attackers can poison an AI model at many different stages in its lifecycle. While training data is certainly one area of concern, there are others: 

• Retrieval engines, especially retrieval-augmented generation (RAG) pipelines that retrieve data from external knowledge bases
• Datasets used for pre-training or fine-tuning that can come from anywhere (including public or shared datasets)
• Code repositories and open-source checkpoints (i.e. model weights) which can come from sites like Hugging Face
• Data generated by users that get fed back into training pipelines
• Third-party tools and APIs that your own model calls into as part of inference
• Content scrapped from the internet that your model relies upon

The manipulation itself can take three forms: injecting new malicious data, altering existing data to skew its interpretation, or deleting data to distort what the model learns from absence.

The Impact of Poisoned Data

Data poisoning limits your model’s performance, typically by targeting the following properties:

• Performance and accuracy. The quality of your model's outputs will decline over time, experiencing higher failure rates. Poisoned data is often concentrated around edge cases and other sensitive use cases where failures are costliest.
• Bias. Carefully crafted adversarial data can cause a model to learn assumptions or draw conclusions that produce output biased toward specific political framings, worldviews, or demographics. Poisoning that causes biased outputs can be hard to spot if many of your model's outputs remain factually accurate.
• Security posture. Poisoned inputs that cause certain behaviors can degrade your model's safety mechanisms, leaving it vulnerable to follow-up attacks that trick your model into producing risky outputs.
• Data privacy. If multiple users are sharing the same model, poisoned datasets structured to increase inter-user data visibility can expose one user's private data in another user's session.

Common Data Poisoning Attack Types

Data poisoning attacks don’t follow a single playbook. At their core, though, all data poisoning attacks influence model behavior. While there are many ways to execute a data poisoning attack, these are among the most common. 

Training Data Poisoning

This is the most prevalent form. Data injected before or during model training is poisoned by:

• Adding biased examples that steer output toward the attacker's desired result.
• Introducing erroneous relationships like benign inputs paired with malicious or incorrect outputs.
• Embedding subtle mistakes that gradually degrade output quality over time.

Attackers can leverage training datasets that come from multiple sources and may consist of thousands of samples. Poisoned examples can easily blend in with benign ones. It doesn't take much poison to cause widespread impact: one 2024 study showed that poisoning just 3% of training data for code-generation models resulted in attack success rates between 12-41% and the generation of code containing vulnerabilities otherwise not present.

“Most of these attacks are fairly easy to mount and require minimum knowledge of the AI system and limited adversarial capabilities. Poisoning attacks, for example, can be mounted by controlling a few dozen training samples, which would be a very small percentage of the entire training set.”

- Alina Oprea, professor at Northeastern University 

RAG and Retrieval Poisoning

Retrieval-augmented generation models are especially prone because they pull in external content during inference time. Ways to attack include:

• Injecting poisoned content directly into your knowledge graph
• Planting poisoned content into websites or other sources your model pulls from
• Planting covert commands in retrieved documents (i.e., indirect prompt injection) to have the model perform attacker provided behavior

To perform a retrieval attack, you don't need access to your training pipeline like you would with a poisoning attack. You just need access to something your model trusts.

Retrieval poisoning and indirect prompt injection attacks differ from typical training-data poisoning attacks. However, they both take advantage of a system's shared assumption that external content is safe. 

Numerous recent reports about AI Security focus on how attacker-controlled content, documents, and external knowledge bases can be used to manipulate model behavior without altering training data. This makes these attack vectors highly pertinent to any business using RAG.

Backdoor Attacks

Backdoors are some of the most dangerous and evasive attack vectors. The attacker causes the model to operate normally under almost all circumstances, but to output some attacker-desired behavior when it receives a trigger (a specific word/phrase, image pattern, or input format). That can be the model divulging private information or ignoring your AI guardrails to cause real damage.

The model will function exactly how you expect during regular testing. Only when presented with the very particular trigger will the backdoor appear. This allows it to slip past evaluation, staging, and perhaps even initial production before being discovered. Hence the need for advanced testing like AI red teaming.

Data Exfiltration via Poisoning

Certain adversaries don't want to make your model fail. Instead, they want to turn your model into a retrieval system. They poison sensitive or formatted data into your training/retrieval pipelines then use extraction techniques during inference time to access it.

Attackers can also attempt to format malicious data in a structured manner that can be triggered/surfaced later on. This risk is compounded when considering multi-user environments. Without proper isolation, one user could unintentionally cause another user’s private data to be revealed.

Supply Chain Risk

Most data poisoning defenses focus on what you can directly control: your training pipeline, your knowledge base, your fine-tuning workflow. But few modern AI systems are trained entirely in-house. They depend on pretrained weights published in public libraries, third-party datasets, open-source software for distributed compute, and APIs that ingest external data at inference time. Each one of those introduces a potential attack surface.

The AI supply chain is where many of the most impactful data poisoning opportunities are. Compromise one checkpoint downloaded from a model hub, and you’ve planted your backdoor on every machine using that checkpoint as a starting point. Compromise one entry in a widely used training dataset, and you’ve planted your backdoor on every model trained with that dataset.

Injection occurs simultaneously at dozens of organizations if that dataset is popular enough. With access to your compute infrastructure, an attacker doesn’t even need access to your data itself.

The issue is transitive trust: By using a third-party component, you trust everyone who previously compiled that component and every library it uses, as well. That's difficult to verify, particularly when your team is iterating quickly.

It can be challenging to vet your third-party components if you lack visibility into what your AI systems actually connect to. Shadow AI (models, agents, integrations deployed outside of formal review processes) increases your supply chain attack surface beneath the visibility of security teams. You can’t know they’ve been leveraged until after they’ve caused problems. The first step to vetting third-party components is knowing they exist.

Mindgard's AI Recon & Discovery helps you map this attack surface. We’ll inventory AI components, enumerate called tools, and surface shadow AI throughout your stack, so you can understand your exposure. Only with proper visibility can you prepare your systems for provenance checks and behavioral testing.

Reducing supply chain risk involves transitioning from implicit trust to explicit trust:

• Scan for provenance prior to ingesting any external dataset/checkpoint. Be aware of who created it, when it was last audited, and which validation it passed.
• Version-pin dependencies and scan for any new disclosures of dependencies you have already deployed.
• Validate pretrained weights yourself before fine-tuning on them. Don't trust a popular repo to be trustworthy.
• Consider your compute infrastructure as part of your attack surface, not just your data.

Teams who ingest the largest variety of third party models and datasets are also likely the ones moving the fastest, importing fresh open-source models and datasets with little to no formal vetting process. Moving quickly and with rigor aren't mutually exclusive, but they do need intentionality around guardrails.

Real-World Examples of Data Poisoning

AI is changing fast, and even the world’s most established companies and models have experienced their fair share of data poisoning attacks. Here are some examples of both direct poisoning attacks and supply chain weaknesses that enable them: 

Microsoft Tay Chatbot (2016)

Microsoft Tay was one of the first examples of data poisoning. Microsoft designed Tay to learn from interactions with Twitter users in real time. It didn’t take long for hackers to discover Tay’s weakness.

Soon after Tay launched in 2016, users realized they could corrupt Tay’s learning process. By pumping Tay with toxic and offensive language, users quickly influenced the chatbot’s behavior. Since Tay’s model integrated user input into its own responses, the chatbot began producing abusive content after only a few hours online. Tay highlighted the danger of allowing training data straight from users to flow directly into a learning loop.

BadNets Backdoor (2017)

In the BadNets research, image- classification models trained without issue until a particular image patch was included in the image. Then the model would produce an attacker-chosen misclassification. This research proved backdoors could test well without detection until presented with the trigger.

Poisoning can affect more than just LLMs was proven with this paper. If you use image, video, audio, or multimodal models your organization needs to consider backdoors and trigger functionality.

Medical LLM Poisoning (2023)

Attackers were able to poison generated AI medical misinformation into The Pile, an open-source dataset used to train LLMs. The insertion of a tiny amount of poisoned data into The Pile measurably increased the rate of toxic medical statements generated by downstream models, demonstrating supply-chain attacks against commonly shared datasets could proliferate poisoning.

The Pile is among the most widely used datasets for training LLMs. While developers use this data to train effective models, poisoned training data can lead to dangerous medical advice.

Hugging Face Repository Compromises (2024)

Hugging Face is one of the largest hosting sites for AI models and datasets. Studies have shown malicious models and models containing backdoors publicly available on Hugging Face repositories.

Due to the common practice of companies pulling pretrained weights and fine-tuned checkpoints directly from public repositories without taking steps to verify these artifacts independently, a single malicious artifact has the potential to seep hidden functionality into dozens if not hundreds of downstream production systems before being discovered.

ShadowRay Attack (2024)

Researchers disclosed a vulnerability affecting Ray, an open-source, widely-used distributed AI compute framework that allowed attackers to execute arbitrary code and poison AI workloads at the infrastructure layer. The report highlights data poisoning attacks can manifest outside of datasets and at the compute layer itself.

Agentic and Multi-Modal Systems

Much of existing work on data poisoning attacks has proceeded from the assumption of a relatively simple target: a lone model which was trained using a single data processing pipeline to produce a single set of predictions.

That is no longer true for a growing fraction of production AI systems.

Agentic architectures (LLMs chaining together tool use/subagent spawning/pushing outputs to other models) change the threat model fundamentally. Poison one node in that ecosystem, and you poison every system that relies on its output.

Imagine a pipeline with multiple agents. A retrieval agent gathers external information, summarizes it, and feeds that summary into a planning agent which acts upon it. If the retrieval agent is poisoned (either because its knowledge base was attacked or via indirect prompt injection through externally gathered content), then the planning agent is fed poisoned input it would have no reason to trust. The downstream agent was attacked without attacking the planning agent itself.

Propagation attacks exhibit two characteristics that make them harder to defend against than traditional single-model poisoning:

• Agents implicitly trust each other. Data sent between agents in most agentic systems is by default considered internal, trusted data. Messages from other agents may not be validated to the same degree as user facing inputs would be.
• An agent's attack surface expands with its capabilities. Give an agent the ability to search the web, look up code repos, run shell commands, query databases, and call external APIs and you've created many injection vectors for poisoned content to enter that agent's processing pipeline.
• Root causes are obfuscated. If you have a system of chained models that produces incorrect or malicious output, you need complete observability into each stage to confidently point to which node was responsible for the failure. Most monitoring and logging solutions aren't designed with this in mind.
• Agents can spawn additional agents. If you have an agent that is an orchestrator spawning child agents, it can be tricked into spawning malicious subagents. You can now run many chains of bad actors at once.

Controls for agentic systems extend typical data poisoning defenses with a few additions:

• Validate at trust boundaries, not just at the edge of the system. Assume the output from one agent is untrusted input to the next.
• Enforce least-privilege at the agent level. If an agent only needs read access, then it should not be able to write. If an agent only needs to summarize documents, it should not be able to execute code.
• Log intermediate outputs through the entire pipeline. Forensic tracing will need access to all nodes, not just final outputs.
• Test pipelines end-to-end with adversarial input. Don’t just test components of a model individually. A node in the pipeline may act differently if it starts receiving malicious inputs from another agent upstream.

The future of data poisoning attacks will likely focus on agentic systems. Autonomy, access to tools, and trust between models mean that one poisoned input can have exponentially spreading consequences that will be more difficult to predict and even harder to stop.

How to Detect Data Poisoning

Data poisoning undermines the very foundations of your AI models. While it can be tough to spot, there are ways to identify the red flags of data poisoning. 

Monitor Model Behavior

To help identify poisoned data, know your model. Track performance metrics such as accuracy and error rates. Benchmarking this information can reveal incremental performance degradation that is often difficult to notice if you don’t track it. 

Degradation can occur naturally, but if there are sudden decreases in performance, this may be cause for alarm. Poisoning attacks are often suspected if there’s a performance slowdown after new updates or data ingestion.

Use Differential Analysis

Performing a differential analysis can help pinpoint where data poisoning may have occurred. Differential analysis looks at what changed between versions of your model. Think of it as a before and after snapshot. Once you know what changed, you can then investigate what data pipeline or dataset could have caused the problem.

Data Provenance Tracking

Data poisoning starts in your data pipeline, so tracking and logging data sources is critical. Make sure you understand where your data comes from, how and when it’s updated, and who can access it. You don’t necessarily need to do this by hand. What’s important is having controls in place to verify data integrity and identify any suspicious activity.

This can be crucial when triaging incidents, as your security teams may need to identify which datasets, third-party vendors, or pipeline stages were the culprit behind poisoned behavior. Studies have found that forensics can help data teams pinpoint poisoned training samples and where attacks entered the AI lifecycle.

Validate Models with Independent Datasets

Regularly test your model with data it hasn't seen before. If your model tests well against training data but poorly or randomly against validation or test data sets (especially with unusual bias behavior), your model may have been poisoned. 

Anomaly Detection

Keep an eye on your pipeline data and model outputs using statistical anomaly detection. Sudden changes in label distributions, token frequency distributions, or clusters of anomalous outputs can all be signs that poisoned data is entering your pipeline. They don’t prove anything on their own, but they’re worth investigating.

The issue with all of the above is time. Baseline monitoring, differential analysis, and anomaly detection only work if you’re actually checking behaviors while they’re occurring. Many teams don’t look at these things until after something has gone wrong. By then, poisoned behavior may have been occurring for weeks.

Mindgard's AI Runtime Threat Detection & Response solution helps solve this issue. Monitoring prompts, responses and agent behavior as it occurs; utilizing contextual guardrails; and taking advantage of intel collected during the previous steps of reconnaissance and red teaming allows an organization to identify and remediate attacks as they are occurring, rather than weeks later. For those teams who already have models deployed into production, implementing runtime monitoring provides the missing link between periodic manual checks and the continuous visibility required for poisoning detection.

5 Best Practices for Preventing Data Poisoning

Data poisoning can affect any LLM. But by layering defenses, you can build resilience against these attacks into every stage of your AI lifecycle. You won’t stop every attack, but these best practices will prevent most exploits and help you identify worst-case scenarios early. 

Red Team Continuously

The best defense is offense. Red teaming your model identifies weaknesses by attacking it in ways a real-world attacker might. This includes:

• Injecting manipulated data into your training pipeline then measuring the downstream impact.
• Searching your model explicitly for backdoor triggers across a wide range of inputs.
• Probing model behavior for edge-case inputs or out-of-distribution (OOD) prompts.
• Examining your RAG pipeline for indirect prompt injection attacks.

As models learn continuously, red teaming can’t be a one-time exercise. Automated red teaming that runs continuously is becoming a necessity for teams that need to maintain velocity.

Apply Zero-Trust Access Control

Data poisoning attacks frequently come down to who has access. It’s easy to give access carte blanche, but doing so is a rookie mistake that leaves your entire network vulnerable. Protect your network with zero-trust principles by: 

• Limiting agentic systems to only the permissions necessary to fulfill their function
• Auditing access logs consistently (which occurs outside of incident response).
• Assuming everything from the outside is malicious until proven otherwise.

Vet All Third-Party Vendors and Data

Unless your data lives entirely in a vacuum, poisoning attacks can come from external sources. This could be from a maliciously crafted integration, or scraped dataset from the wilds of the internet. However, your engineers will still rely on third-party APIs and shared model repos to build applications at speed. Before using any third-party dependency, you should:

• Get in contact with the vendors directly to learn how they validate and trust their data and models.
• Manually evaluate pretrained weights with known backdoor detection methods prior to fine-tuning on them.
• Keep an eye on shared repos for newly published vulnerabilities and keep track of the versions you've used.

Isolate User Data from Training Pipelines

Do not automatically include user-supplied data in training flows. Segregate inference-time data from training data, and validate and manually review user-supplied data before using it for model updates.

Prepare a Mitigation Playbook

Expect your defenses to be breached. Have a plan for remediation ready to go:

• Isolate poisoned data sources.
• Analyze downstream dependencies that ingested poisoned data.
• Retrain from a clean checkpoint. Never expect deleting malicious data to undo learned behavior.
• Discard poisoned datasets. Even if you cleaned your dataset, you should not reuse it.
• Record what happened with sufficient detail to improve your detection and blocking controls.

Cleaning your dataset of poisoning does not negate the data your model has already trained on. You may need to completely rebuild your model from clean data if poisoning is severe.

When to Trust a Pretrained Model

Auditing or “vetting your third-party models” is advice that shows up in just about every AI security checklist, including this one. The details of how to do it effectively are seldom discussed. Here are some recommended practices you can adopt when evaluating a pretrained model/checkpoint before adding it to your pipeline:

Establish Provenance

Before you even run your first inference, ask yourself:

• Who created it? Is there a well-known institution behind it that you can research? Treat code uploaded under a pseudonym or anonymously to public code repos with far greater scrutiny.
• How was it created? Was there a model card written that documents where the training data came from, how it was processed, and what known weaknesses it has? While lack of a model card isn’t a death sentence, it does place more onus on you to verify through testing.
• Was it reviewed? Has the model been through any third-party auditing, especially if released by a university or larger lab? Audits are far more trustworthy than self-disclosed safety information.
• What is the commit history? Is there a public changelog for the model you are using? Models that have public commit history are far easier to reason about than a magically created model with no backstory of improvements/changes.

If you cannot answer these questions with any degree of certainty, you should consider the model to be high-risk until you can

Check for Known Bad Artifacts

Perform some passive checks against a list of known bad artifacts before dynamic behavioral testing begins:

• Search the model’s repo page, paper, and recent CVEs (Common Vulnerabilities and Exposures) for discussions around identified vulnerabilities or poisoning attempts.
• Search to see if the model has been referenced in any published lists of backdoored/artifacts that have been compromised. There are researchers at CMU, MIT, Northeastern regularly releasing papers detailing attacks on specific models and checkpoints.
• Search the repo issues and PRs for any suspicious activity. Look for things such as edits to weights/config files without proper explanation in the last few commits.
• Verify any provided checksum/hashes match those published by the authors. This can confirm you didn’t receive a tampered-with artifact.

Run Behavioral Testing

Static provenance tests can tell you where your model came from. Behavioral tests tell you how your model behaves:

• Baseline evaluation: Run your model on a known benchmark in your domain and verify reported vs. observed performance. Unexpected deviations are a reason to look deeper.
• Out-of-distribution probing: Feed the model examples it should not have seen during training and check whether outputs gracefully fail or stay unexpectedly safe.
• Bias/consistency checks: Feed semantically identical phrases which have different demographic/ political framings or contain sensitive subject matter. Failure to return consistent results can be a sign of poisoned examples.
• Edge case stress testing: Create inputs designed to trigger likely attack vectors (e.g., extremely long inputs, strangely formatted inputs, sensitive topic combos). Backdoored models typically perform normally on clean inputs but may misbehave on crafted edge cases.

Performing these tests manually works for an initial one-time intake evaluation. But a model gets updated, fine-tuning affects behavior, and novel attack strategies are constantly being developed. 

Mindgard's AI security platform was designed with this problem in mind. It automatically red teams your models, even pretrained checkpoints that your team is testing at any given time. It exposes the anomalies that point in time manual testing would never catch. For companies using third party models regularly, automated behavioral testing is the only way intake can become streamlined vs. bottlenecked.

Use Automated Backdoor Scanning

There are multiple tools made specifically for scanning models for backdoor signatures. These aren't guaranteed to find every backdoor, but they provide a layer of automated testing that can catch issues that manual testing won't:

• Neural Cleanse (Wang et al., 2019) scans for possible backdoor triggers by attempting to find localized input deltas that lead to large output changes.
• STRIP (STRong Intentional Perturbation) (Gao et al., 2019) attempts to detect whether backdoored inputs are being fed to your model at inference time by checking how consistent predictions are when exposed to superimposed noise.
• ABS (Artificial Brain Stimulation) (Liu et al., 2019) scans your model for neurons that fire excessively under certain trigger inputs.
• Trojan Detection via Meta Neural Analysis (Xu et al., 2019) trains a meta classifier to differentiate clean vs. trojaned models given their learned weights.

The tactics described above all share the same principle: analysis tools scan model weights and intermediate activations for statistical signatures that reveal backdoor triggers were used during training. Detection is strong for known attacks but weaker for detecting novel or advanced trojans that do not leave known signatures. Behavioral testing helps solve this problem.

Behavioral testing (as opposed to weight-scanning tools that analyze the model) tests how the model behaves when attacked. By chaining attack primitives together in plausible attack chains, adversarial testing can reveal surprising behavior that static inspection would never catch.

Mindgard's AI Risk Discovery & Assessment solution performs this type of testing, repeatedly exercising models with an advanced attack library, and verifying guardrails hold under realistic attacker behavior. To fully intake assess a pretrained model, run weight-scanning utilities in parallel with behavioral assessment to cover as many vectors as possible: one exposes known signatures embedded in model architecture, the other exposes how the model will actually behave when exploited.

Stage Before Deploying to Production

Even if your model has passed all of the above tests, it should still be staged prior to deployment to production:

• Deploy the model to an environment that feeds it realistic, but non-production traffic.
• Pull metrics on the outputs of your model and compare them closely to your baseline for the first several days/weeks of staging.
• Restrict the model's access to sensitive data and downstream systems while it's in staging.
• Have clear criteria for when the model should be promoted to production. Avoid allowing indefinite retention in staging without forcing or prompting for decision making. 

A Word of Caution: Popularity Does Not Mean Safe

Downloads, stars, or cool lab banners do not negate the need for evaluation. The Hugging Face examples mentioned previously exploited models hosted in some of the most popular repositories. Just because a project is popular does not mean it is safe. Assume nothing about a model’s intentions based on where it’s from. Google’ official lab could host dangerous models just as much as a name you’ve never heard of. Once it’s in your pipeline, you’re just as exposed.

The Threat You Don’t See Coming

Many organizations focus mostly on easily observable failures: jailbreaks, prompt injections, blatant hallucinations. Data poisoning can occur without producing failures that you’ll easily see. Poison subtly corrupts what your model knows, often quietly over time, long before anyone realizes anything’s wrong with how it responds.

Which means prevention is key. The controls you want to have are the ones you have before poisoning can occur: robust data provenance, tight access controls, continuous testing, third party vetting. Detection and response are useful, but come downstream.

It only gets worse as our AI systems become more distributed. They’re sharing datasets and checkpoints and external retrieval sources with other teams and organizations. If you’re training and putting AI models into production you should understand where your model’s training data is coming from and what controls exist around it.

Mindgard’s AI Security Platform empowers developers to discover blindspots before hackers do by surfacing behavior standard validation tests miss. Learn how your models will perform when under attack. Schedule a demo today to learn how continuous red teaming can help you secure your AI systems.

Frequently Asked Questions

What's the difference between data poisoning and model drift?

Model drift occurs naturally as data evolves. Data poisoning is intentional. It's crafted to change how a model behaves in ways that advantage the attacker. This makes poisoning far more pernicious and difficult to detect.

What are the most common signs of data poisoning?

Look for slow decreases in accuracy or consistency, inexplicable output bias, triggered behavior after a data ingestion event, or seemingly-normal outputs that generate particular wrong answers to particular inputs. The latter is the hallmark of a backdoor attack.

Does data poisoning only happen during training?

Not necessarily. Training data is by far the most discussed vector, but poisoning can happen whenever your model takes in external input. That includes your retrieval systems, fine-tuning pipelines, third-party plugins, etc. RAG systems are especially at risk as they retrieve live external content during inference.

What does red teaming specifically look for regarding data poisoning?

Red teaming looks for behavior that traditional validation will not catch: covert backdoors, trigger-laden behavior, vulnerabilities that only present themselves during adversarial use. Red teaming is one of the only ways to discover backdoor attacks prior to production activation.

Platforms like Mindgard implement continuous red teaming to allow you to discover these problems before they become larger issues.