Membership inference attacks can reveal whether sensitive records were used to train AI models or appear in RAG retrieval stores, making targeted privacy testing, reduced overfitting, differential privacy and access controls essential defenses.
In 2017, researchers at Cornell Tech demonstrated that using queries to a model hosted on Google’s prediction API, they could learn with 93.5% accuracy whether a particular shopping record was part of its training set or not. The researchers did not see the training data nor the model’s parameters. They just queried the model and observed its confidence.
That’s a membership inference attack. Presented with a data record and query access to a model you build, the attacker determines if that record was in your training set. If you trained your model on hospital discharges, or HIV status or your company’s customer list, then the answer to that yes/no question is the sensitive data itself.
Eight years later, that attack went from academic paper to international regulation. In December 2024, the European Data Protection Board adopted Opinion 28/2024, which states that a model is only considered anonymous if the likelihood of extracting personal data is insignificant, assessed against “methods like membership inference or inversion attacks.” A research attack is now a GDPR test.
This guide walks through how membership inference attacks work and where they can succeed, where researchers have found them largely ineffective (such as pretrained LLMs), how OWASP, MITRE ATLAS™ and NIST map to privacy risks, and what controls limit the leak. The bottom line up front is this: Your fine-tuned models and your RAG retrieval store are the risk, not the LLM base model, so you should test there.
MINDGARD
Answer seven questions about one model, fine-tune or RAG pipeline. The scorer estimates exposure, names the attack an adversary would reach for first and tells you which control to fix before you run the test.
In a membership inference attack, a person who can query a machine learning model attempts to establish whether a given record was included in the data used to train it. According to NIST AI 100-2 E2025, membership inference attacks involve inferring "the presence of data in the training set," and the publication classifies this attack as NISTAML.033 under data privacy attacks that apply to both predictive and generative AI models.
An attacker will typically possess three pieces of information: The record being queried, query access to the model, and some prior knowledge of the training data's distribution. The attacker’s output is a binary true/false decision, often with an associated confidence score, for each queried record.
The underlying idea behind membership inference attacks isn’t new. It actually predates machine learning. Back in 2008, Homer et al. was able to infer if an individual was part of a genome-wide association study using only aggregate allele frequencies that were published. As a result, the US National Institutes of Health actually removed aggregate GWAS data from public access. Shokri was able to apply this same concept to machine learning models in a 2017 paper. The difference is that the “aggregate” is now the trained model itself, and the query is made through the API.
Membership inference doesn’t leak anything about the model. It leaks information about people.
Shokri’s team tested the attack against a model trained on 67,330 hospital discharge records from Texas patients to predict each patient’s primary medical procedure. They reached per-class precision above 0.7 for half the procedure classes, above 0.85 for more than 20. If the attack returns a positive result, it proves an attacker's specific patient’s record, with diagnoses and procedure, was used to train the model. That lets attackers know that the individual was hospitalized, which could be sensitive health information by itself.
This risk becomes even more acute if a model is trained on records from a single population. If attackers know a model was trained only on records from people with a specific diagnosis, then membership inference is equivalent to learning that diagnosis.
The same logic holds for a churn model trained on your customers, a fraud model trained on labeled fraudulent accounts or a customer support chatbot fine-tuned on real support tickets. If your model’s training data is filtered on a particular attribute, membership is a proxy for that attribute.
Four inference attacks all go through the same prediction API, and their names get conflated in vendor threat models. This table distinguishes them by what the attacker ultimately acquires.
Membership inference and extraction go hand-in-hand: a model extraction attack leaves the attacker with a free offline copy which can be used to train the membership attack against. ATLAS™ names AML.T0005.001 Train Proxy via Replication as the technique attackers use to build shadow models. Mindgard's overview of adversarial attacks covers the broader family.
All membership inference attacks leverage one observation: A trained model behaves differently on data it has seen during training versus data it has not. The behavior difference can be observed through lower loss, higher confidence, lower perplexity or some other pattern in the internal activations.
Shokri's research, described in more detail below, showed that attack precision correlates with the train-test accuracy gap. With two classes and a gap of .015, they achieved 0.505 precision (a coin flip). With 100 classes, a gap of .34 and training accuracy of .999, they achieved 0.935 precision.
Overfitting doesn’t tell the whole story (Google’s less-overfitted model leaked more than Amazon’s), but it is a good first thing to quantify.
There are other differences in these attacks based on how they quantify the gap and how much access they require.
Shokri’s attack must first overcome one limitation: The attacker doesn’t have access to the target’s training data. This means it knows nothing about the target's behavior on member versus non-member records. So the attacker must construct this knowledge independently. It trains multiple shadow models designed to behave like the target, feeding them data drawn from the same distribution used to train the target, and (if possible) obtained from the same machine learning service. Each shadow model is trained on data chosen by the attacker, who therefore knows for sure which records were members and which weren’t.
Next, the attacker queries its shadow models with both members and non-members, recording their confidence scores. These labeled examples teach another machine learning model (one called an attack model) to recognize the difference between a member’s and a non-member’s confidence score patterns. When it can do so with high enough accuracy, the attack model is unleashed on the intended target. For CIFAR, Shokri’s team trained 100 shadow models; for Purchase, 20; and for Texas, 10.
The shadow data doesn’t even have to be real. The attacker can generate it by starting with random records and repeatedly adjusting them, keeping any change that makes the target more confident. Records the target is very sure about tend to look like its training data. Shadow models trained on these synthetic records still reached 0.896 precision on Purchase-100 (the 100-class version of the shopping dataset), close to the 0.935 the attack achieved with real data.
Yeom’s attack simply abandons the shadow models. Instead, it depends on overfitting: a model’s loss is typically lower on records it trained on than on new records. The attacker computes the target’s loss on a record and declares it a member if the loss is below a threshold. The attack was cheap, and thus became the baseline each new paper attempted to improve upon.
The weakness of that attack only shows when you pick the right number. When Nicholas Carlini and colleagues re-ran it on CIFAR-10 for the paper that introduced LiRA (below), it achieved 59.4% balanced accuracy, slightly better than random guessing. But at a false positive rate of just 0.1%, the attack correctly identified 0.0% of dataset members. The attack can learn a general trend dataset-wide, but it can’t say for sure that any given record was used during training. The discrepancy between these two statistics was what motivated new standards for reporting attacks.
Membership Inference Attacks From First Principles reset the bar. Carlini et al. showed that average accuracy isn't indicative of whether an attack can be certain of any given member. An attack might be right 60% of the time overall, yet never be confident about any single prediction it makes. Privacy damage occurs on a per-person basis. Instead, they suggest evaluating attacks on their true positive rate (TPR), or percentage of actual members identified by the attack, at false positive rates (FPR) of less than 0.1% (i.e., less than one non-member in a thousand is incorrectly flagged).
LiRA addresses the blindspot in Yeom’s single-threshold attack: if a record is easy for any classifier to classify correctly then a low loss will tell you nothing about whether it was used in training. LiRA calibrates individually for each record. It trains 256 shadow models, each on a random split of the dataset such that each record has been seen by about half the shadow models and unseen by the rest. For each record the attacker is interested in, it compares confidence scores from the two sets of models to infer what “member” and “non-member” look like with regards to that record specifically. It then determines which group the target model’s score more closely resembles.
LiRA is able to identify 8.4% of members at a false positive rate of 0.1% on CIFAR-10. The closest comparison from prior work achieves 2.2%. The paper’s headline claim is up to 10x more power at the same low false positive rate. The cost for this improvement is compute: training 256 shadow models is expensive. Imitative MIA, presented at USENIX Security 2026, demonstrates similar results at less than 5% of the compute by training just a handful of target-informed shadow models instead of hundreds of independent ones.
If the API returns only a predicted class, the attacker probes robustness rather than confidence: how much can you change a record before the label changes? Members lie further from the decision boundary. That's why hiding confidence scores impedes the attack but does not defeat it.
When it comes to language models, text is the record itself, but the signal is the probability the model assigns to each word (technically each token, sometimes just a fragment of a word). Shi et al. observed that text from outside a model's training set will generally have some tokens whose likelihood the model predicts to be extremely low, while texts from the training set will not, as the model has already adjusted its weights to account for the odd tokens they contain.
Min- K% Prob simply identifies the k% of tokens with lowest likelihood in a given text, averages their likelihoods, and declares the text a member if the result is higher than a fixed threshold. Like the others above, it requires no shadow models or training.
Results for this and subsequent attacks are presented as AUC (area under the ROC curve), which quantifies an attack's ability to distinguish members from non-members across all classification thresholds. An AUC of 0.5 corresponds to random guessing and 1.0 to perfect. On the WikiMIA benchmark, Min- K% Prob achieved an average score of 0.72 AUC compared to 0.67 for a simple baseline that assigns each text a score based on how surprising it found the text to be overall. (Details on pretrained LLMs below explain why WikiMIA scores are artificially high.) Evaluated against GPT-3 and 100 copyrighted books from the Books3 corpus, Min- K% Prob achieved 0.88 AUC. It identified more than half of the passages evaluated as training data for nearly 90% of those books.
Many LLM deployments use retrieval-augmented generation, or RAG. Rather than answering questions based solely on what got baked into its weights at training time, the system will query a database of documents (support tickets, clinical notes or company knowledge bases, etc.), paste the most relevant ones into the prompt given to the model, and then let it respond. The database of retrievals is a brand-new surface for private information exposure.
You don’t even need encryption-breaking hardware to attack it. Just a chat window will do. Anderson, Amit and Goldsteen (IBM Research, 2024) asked the LLM, in plain language, whether a particular passage of text was contained in its retrieved context. Their best prompt was simply “Answer with Yes or No.”
There are two threat models here. Are you allowed to know the confidence of a yes prediction? In black-box mode, you only see whether the system says yes or no. In gray-box, you get the probability it assigned to yes. This gives you a much finer-grained signal about what it knows. The attack achieved 0.89 AUC in the black-box setting and 0.96 in the gray-box against Llama-3-8B-Instruct running on HealthCareMagic, a popular LM dataset made up of conversations between doctors and patients. The true positive rate at zero false positives was 0.73. Overall averages across three models and two datasets were 0.80 and 0.90.
The model didn’t need to have memorized anything. The retrieved documents sat in its prompt, so it could simply read them and say whether the passage was there. The retrieval store, not the model, was the leak.
Against a pretrained LLM, mostly no. Against anything built on top of one, yes.
That held up when Duan et al. (2024) put it to the test on Pythia, a full lineup of open models spanning 160M to 12B parameters, each trained on the Pile, another publicly available dataset. This gave them a clean test because the training data is public and they knew which texts were members ahead of time. They ran five attacks including the loss threshold attack and Min-K% Prob attack described above. LossThresh did poorly. For the 12B model, AUC ranged from 0.475 to 0.582 across Wikipedia, Pile CC, PubMed Central, ArXiv, DM Math and HackerNews. That’s right around the AUC of 0.5 for a coin flip. Only on GitHub was it significantly higher at 0.690. Their title asks if MIAs work on LLMs, and their answer for pretraining data is that they “barely outperform random guessing.”
Why is it so hard? The authors give two reasons. First, because a model sees each example only once across hundreds of billions of tokens during pretraining, it never overfits to any given text, and overfitting is the signal every attack looks for. Second, there’s a lot of overlap between members and non-members. For non-member Wikipedia and ArXiv texts, over 30% of seven word phrases also appeared in member texts. In other words, the model had effectively seen most of what was supposed to be “unseen” data already.
The same paper explains why other studies report much stronger results. Benchmarks like WikiMIA use Wikipedia articles written before a model’s training cutoff as members and articles about later events as non-members. The attack can then tell the two apart by their newer topics and vocabulary, whether or not it detects membership. When Duan et al. drew non-members from a later period in this way, the AUC of one attack on Wikipedia jumped from 0.579 to 0.796. The attack was detecting a date, not membership.
Meeus et al. (2024) made this argument in depth. They tested six LLM membership inference benchmarks that had been created after the fact by scraping for presumed members and non-members. For each of the six datasets, they could perfectly separate member from non-member text with a simple classifier that looked solely at which words appeared in the text, without ever querying the LLM. Those benchmarks measure differences between the datasets, not membership. Newer attacks such as AttenMIA (January 2026, 0.996 AUC on WikiMIA-32) should be read with that caveat until they’re tested on randomized splits, where members and non-members are drawn at random from the same pool before training so membership is the only difference between them.
The picture reverses as soon as the model is adapted to a small dataset.
This is the practical conclusion for a security team. The base model you licensed is the least exposed part of your stack. The customer data you fine-tuned on and the documents you indexed for retrieval are where a membership inference test will find something.
Four consequences follow from a working attack, and three of them are legal rather than technical.
EDPB Opinion 28/2024 names membership inference as one of the attacks a model must resist to count as anonymous. If a model fails that test, it may itself contain personal data, and GDPR applies to the model, not just its outputs, including the right to erasure. In health and financial applications, a successful attack is a privacy breach in its own right. If a model was trained only on records of people with a particular diagnosis, debt problem or fraud flag, proving that someone’s record was in the training set proves they have that diagnosis, debt or flag.
IBM’s 2026 Cost of a Data Breach report puts the average cost of a model inversion incident, the closest category it measures, at $6.07 million. IBM does not break out membership inference separately.
Min-K% Prob’s book study is a membership inference attack used as evidence. The ability to show that a specific copyrighted work was in a model’s training set, with a stated AUC, is the kind of result that appears in discovery.
The same tooling supports benchmark contamination audits, where Shi et al. reached 0.86 average AUC detecting downstream test sets in training data.
If a user invokes a right to erasure and you retrain or unlearn, the membership inference attack is how you prove the record is gone. Shi et al. list privacy auditing of machine unlearning as a direct application, and Meeus et al. recommend injected canary sequences as the sound way to measure it.
A membership oracle converts an otherwise blind training-data extraction attack into a targeted attack. In research conducted in December 2025 about targeted extraction from LLMs, membership inference is used to determine which candidate generations are in fact members of the training data. Step one is confirming membership. Step two is regenerating the record.
The attack has a home in every major AI security framework, which makes it easy to scope and to report.
The OWASP Machine Learning Security Top 10 lists it as ML04:2023 Membership Inference Attack. Note that its one-line description says the attacker “manipulates the model’s training data,” which describes poisoning, not inference. The attack scenario and the five prevention measures (shuffled training data, obfuscation or differential privacy, regularization, reduced training data, testing and monitoring) are the useful parts.
For LLM applications, the risk falls under LLM02:2025 Sensitive Information Disclosure in the OWASP Top 10 for LLM Applications.
ATLAS™ files it as AML.T0024.000 Infer Training Data Membership, a sub-technique of AML.T0024 Exfiltration via AI Inference API, alongside inversion (.001) and extraction (.002). The entry notes that shadow models may come from AML.T0005.001 Train Proxy via Replication. Mindgard maps its attack library to ATLAS™ through the MITRE ATLAS™ Adviser.
NIST AI 100-2 E2025 classifies membership inference as a data privacy attack (NISTAML.033) in both the predictive and generative AI taxonomies, next to data reconstruction, attribute inference and property inference, and separate from model extraction, which it treats as a model privacy attack. In the NIST AI Risk Management Framework, the controls are found under Measure and Manage.
No control makes membership inference impossible. Any model that generalizes from data retains some signal about it, and label-only attacks show that hiding confidence scores only slows the attacker down. The goal is to push the true positive rate at low false positive rates toward chance, then measure it.
Six controls do most of the work, in rough order of effect.
Run the attack yourself, against the fine-tuned model and the retrieval store, with randomized member and non-member splits or injected canaries so the result measures membership rather than distribution shift. Report true positive rate at 0.1% and 1% false positive rate, not accuracy. An attack with 60% accuracy can still name individual members with certainty, and an attack with 0.1% TPR at 0.1% FPR is harmless at 90% accuracy.
Track the number across releases the way continuous AI pentesting tracks jailbreak success rates. A new fine-tune, a new index or a new output field each reopens the question.
Membership inference belongs in the same AI red teaming scope as prompt injection, jailbreaks and extraction, not in a separate privacy review. Most of the AI red teaming tools worth shortlisting either ship a membership inference module or accept the shadow-model and Min-K% scripts from the papers above. Use the risk scorer at the top of this post to estimate where your deployment sits before you start testing.
Membership inference is in scope for Mindgard’s AI red teaming services and AI penetration testing services alongside extraction, inversion and training-data leakage, for foundation, fine-tuned and open-weight models and for the RAG pipelines and agents built on them. Findings map to OWASP ML04 and LLM02 and to MITRE ATLAS™ AML.T0024.000, with the TPR-at-low-FPR evidence a privacy or legal team can use.
The same attacks run in the platform’s attack library on every build through automated AI red teaming, so the number moves when the model does. Book a demo to see what your fine-tune and your retrieval store give away.
A membership inference attack is an attack in which an adversary with query access to a machine learning model determines whether a specific data record was part of its training set. It was formalized by Shokri, Stronati, Song and Shmatikov in 2017, who reached 93.5% precision against a model hosted on Google’s prediction API. The attack exploits the fact that models behave more confidently on data they were trained on.
Usually black-box. Shadow-model, loss-threshold, likelihood-ratio and label-only attacks need only the model’s outputs, and the prompted attack against RAG systems needs only a chat interface. White-box variants such as AttenMIA use internal attention activations and report higher AUC, but they require access an external attacker rarely has.
Against pretraining data, barely: Duan et al. (2024) measured AUC between 0.475 and 0.582 for most domains against a 12B-parameter model, close to the 0.5 of random guessing, because pretraining sees each example about once. Against fine-tuned models, RAG retrieval databases and in-context data the attacks work well, with reported AUC from 0.80 to 0.99.
Membership inference answers whether a known record was in the training set. Model inversion reconstructs training data the attacker does not already have, from the model’s outputs. MITRE ATLAS™ lists them as sibling sub-techniques, AML.T0024.000 and AML.T0024.001, under Exfiltration via AI Inference API.
Reduce overfitting (fewer epochs, regularization, deduplication), train with differential privacy for sensitive data, restrict outputs to labels or rounded probabilities, harden RAG prompt templates, rate-limit repeated probing and keep sensitive records out of training sets where possible. Then measure the attack yourself and track true positive rate at a low false positive rate across releases.
EDPB Opinion 28/2024 says an AI model is only anonymous if personal data cannot be extracted from it with more than insignificant likelihood, and names membership inference and inversion attacks as the methods to assess against. A model that fails the test contains personal data, so the GDPR applies to the model and to the rights of the people in its training set.
Mindgard secures AI models, agents, and applications against exploitable threats. Spun out of more than a decade of pioneering AI security research at Lancaster University, Mindgard combines offensive security expertise, AI red teaming, attacker intelligence, and automated protection to help organizations discover, assess, and mitigate risk across AI models, agents, and applications.
We found vulnerabilities in AI from OpenAI, Meta and AWS. See how Mindgard finds and proves the weakness attackers could exploit in yours.
