Model extraction attacks use AI outputs to cheaply replicate capabilities, creating intellectual property, privacy, and security risks that require layered defenses, behavioral monitoring, and regular extraction testing.
In February 2026, Anthropic announced that three AI labs had extracted over 16 million queries from Claude using approximately 24,000 fake accounts, then trained their models on the outputs they received. No server was hacked. Each of those millions of queries was sent via the public API just as a paying customer’s would be.
That’s a model extraction attack. Given nothing more than query access to a deployed model, an attacker sends inputs and uses the input/output pairs they receive back to train a local model to mimic the original.
The attack isn’t new. Tramèr and colleagues demonstrated it against BigML and Amazon ML in 2016. It’s been known since as model stealing, model theft, and lately, a distillation attack to reflect its utility against large language models.
What is new is how often it happens and how much is at risk. According to IBM’s 2026 Cost of a Data Breach report, model inversion attacks (the closest category they survey) now cost $6.07 million on average, making them the costliest AI-related incident they track.
In this guide, we’ll dive into how extraction attacks work, what they’ve cost in publicly-reported breaches (including one $50 clone of a ChatGPT task Mindgard performed), how groups like OWASP and MITRE ATLAS™ are categorizing them and what controls can increase an attacker’s costs.
MINDGARD
Answer seven questions about one deployed model or agent API. The checker assesses your exposure across seven areas and identifies the gap an attacker would use first.
Model extraction attacks involve rebuilding a functionally equivalent machine learning model simply by querying an existing one and observing its outputs. Attackers can pull this off without access to the training data, the architecture, or the parameters. The API is the vulnerability.
The 2016 paper that named the problem, Stealing Machine Learning Models via Prediction APIs, showed near-perfect fidelity copies of logistic regression, neural network and decision tree models hosted on commercial prediction services. The authors also found that hiding confidence scores did not stop the attack, only slowed it.
The attack has picked up several names since then. They mostly describe the same thing, but knowing them all helps when you’re reading a vendor’s threat model or a framework entry and want to be sure it covers this risk.
These three terms are interchangeable. RAND’s AI threat landscape guide describes the attack as replicating “a deployed model’s functionality by systematically querying it,” and rates the consequence as high and ranks the consequence as high due to loss of IP, lost revenue, and exposure to follow-on attacks via the surrogate.
Distillation can be done legitimately when you train a compact student model on the outputs of a large teacher that you own. It becomes an attack when the teacher is someone else’s and your queries violate their terms of service. Google’s Threat Intelligence Group refers to the two terms interchangeably, noting that model extraction attacks “are often called ‘distillation attacks.’”
Targets of the 2026 wave of distillation attacks against Claude and Gemini included reasoning. Attackers prompted the models to articulate their chain of thought step-by-step, because reasoning traces are the most valuable type of training data that a frontier model can produce.
Model extraction is one of three inference attacks that work through the prediction API. The others target the training data rather than the model.
For the wider family of attacks these sit in, see Mindgard’s breakdown of adversarial attacks and their consequences.
All extraction attacks run through these four steps, whether they’re a 2016 decision tree or a 2026 distillation campaign.
The variants differ in what step one targets.
The attacker recovers some of the model’s actual weights, not just an approximation of its behavior. In Stealing Part of a Production Language Model (Carlini et al., 2024), researchers from Google DeepMind, ETH Zurich and partner institutions recovered the full embedding projection layer of OpenAI’s Ada and Babbage models for under $20 in API calls. This is the final layer of the model, which turns its internal representation into a probability for each possible next token. They also confirmed the hidden dimension of gpt-3.5-turbo, meaning the size of that internal representation, a detail OpenAI had not made public. They estimated the full projection matrix of gpt-3.5-turbo would cost under $2,000 to extract.
This was the first attack to pull precise, nontrivial internal information from a black-box production LLM. It relied on the logit bias parameter, an API feature that lets users raise or lower the likelihood of specific tokens appearing in a response. By combining it with the probability information the API returned, the researchers could work backward to the layer’s weights. OpenAI and Google changed how their APIs handle this after disclosure.
The attacker clones a single capability, such as question answering, into a much smaller model rather than copying the whole thing. The result is cheaper to run and can be studied offline to find weaknesses in the original. Mindgard’s research team did this in Model Leeching (Birch, Hackett, Trawicki, Suri and Garraghan, 2023), extracting ChatGPT-3.5-Turbo’s question-answering ability for about $50 in API costs. The full study is covered below.
The attacker harvests millions of outputs across many tasks to lift a frontier model’s general capability into a competitor. The Anthropic and Google cases from February 2026 are the documented examples, covered below, and they are why “distillation attack” has largely replaced “model extraction” as the term most people use.
A model’s step-by-step reasoning is some of the most valuable material a distiller can collect, because it shows how the model reached an answer, not just the answer itself. To protect it, providers now return reasoning as a summary or as an encrypted block: an opaque package the developer can’t read but can pass back to the API so the model can pick up where it left off. Panfilov et al. (August 2026) found that these encrypted blocks were interchangeable across models from the same provider.
That gap gave attackers a workaround. The researchers took an encrypted trace from a strong, heavily safeguarded model and fed it into a cheaper, less protected sibling, which decoded it and wrote it out in plain text. Using the same method on agent session logs that developers had shared publicly, they decoded 315,320 reasoning blocks from 6,708 sessions. Providers have since tightened how these blocks are handled.
The attacker learns a model’s architecture by watching the hardware it runs on, not its outputs. In DeepSniffer (Hu et al., 2020), researchers recovered full neural network architectures by monitoring GPU memory access patterns while the model ran. The architecture alone isn’t the model, but it gives the attacker a blueprint, which makes it easier to train an accurate copy or craft inputs that fool the original. This is the one route in this list that doesn’t go through the API. It requires access to the hardware, which makes it most relevant to models running on edge devices or shared infrastructure.
The attacker learns a model’s architecture by watching the hardware it runs on, not its outputs. In DeepSniffer (Hu et al., 2020), researchers recovered complete neural network architectures from the volume of memory reads and writes on a GPU while the model ran, with no prior knowledge of the model. The architecture alone isn’t the model, but it gives the attacker a blueprint: with it, DeepSniffer’s adversarial attacks succeeded 75.9% of the time, up from 14.6% to 25.5% without it. This is the one route in this list that doesn’t go through the API. It relies on monitoring the hardware directly, through side channels or by snooping on the memory bus.
Three published cases set the price range.
In 2023, Mindgard’s founding research team at Lancaster University ran an extraction attack against ChatGPT-3.5-Turbo. The Model Leeching paper distilled the model’s question-answering capability on SQuAD into a student roughly 100 times smaller.
Total API spend: $50. The student reached 73% exact-match agreement with ChatGPT’s answers, and scored 75% exact match and 87% F1 on SQuAD in its own right.
The second half of the experiment is the part defenders should read twice. The team crafted adversarial examples against the cheap clone, then fired them at the real ChatGPT. The transferred attack lifted the success rate against the target by 11 percentage points.
That is why a partial copy is not a partial problem. Extraction gives the attacker an unlimited, free, offline range to rehearse on. Mindgard’s earlier PINCH framework reached the same conclusion across 21 architectures: stolen models can match the victim’s accuracy with very different internals, so accuracy alone is a poor measure of how much leaked.
The 2024 Stealing Part of a Production Language Model result put a dollar figure on parameter extraction from a live commercial API: under $20 for Ada and Babbage’s full projection matrices, and under $2,000, by the authors’ estimate, for gpt-3.5-turbo’s.
The enabling features were ordinary ones that developers use every day. The API returned top-k log probabilities, the model’s confidence scores for its most likely next tokens, and accepted logit bias, which lets a caller push particular tokens up or down. Combined, they let the researchers collect the model’s full output scores for thousands of prompts. Because those scores are all produced by the same final layer, analyzing them together revealed the size of the model’s hidden dimension and the weights of that layer, up to a mathematical transformation that doesn’t affect what it does.
OpenAI and Google changed their APIs after disclosure. OWASP’s Top 10 for LLM Applications responded too: the 2025 edition added “Limit Exposure of Logits and Logprobs” as a named prevention and cited this paper, and the 2026 edition keeps logit and logprob exposure under Unbounded Consumption, now ranked LLM06:2026.
Anthropic’s February 2026 disclosure attributed more than 16 million exchanges with Claude, run through roughly 24,000 fraudulent accounts, to three labs: DeepSeek at more than 150,000 exchanges, Moonshot AI at more than 3.4 million and MiniMax at more than 13 million. All three went after the same things: agentic reasoning, tool use and coding.
Because Anthropic doesn’t sell access to Claude in China, the labs went through commercial proxy services running what Anthropic calls “hydra cluster” architectures: large networks of fraudulent accounts spread across Anthropic’s API and third-party cloud platforms, where each banned account is simply replaced by another. One proxy network managed more than 20,000 accounts at once and mixed distillation traffic with ordinary customer requests to make it harder to spot.
Anthropic caught the MiniMax campaign while it was still running, before MiniMax had released the model it was training. When Anthropic shipped a new model mid-campaign, MiniMax redirected nearly half its traffic to the new system within 24 hours. The attackers were watching the release calendar more closely than most customers.
Google’s GTIG AI Threat Tracker (February 2026) described a separate campaign from the same period: more than 100,000 prompts aimed at getting Gemini to reveal its full reasoning. Gemini normally summarizes its reasoning before showing it to users, so the attackers framed their request as a formatting rule, instructing the model that “the language used in the thinking content must be strictly consistent with the main language of the user input.” The breadth of the questions suggested an attempt to replicate Gemini’s reasoning in non-English languages across many kinds of tasks. Google says it detected the campaign in real time and protected the reasoning traces.
The two reports point defenders in the same direction: watch for extraction patterns in API traffic, not just individual bad prompts. Anthropic makes the point most directly. Its example prompts looked harmless on their own, and the campaigns only became obvious when the same narrow request arrived tens of thousands of times across coordinated accounts.
Extraction is often filed under intellectual property risk and left to legal. It ranks among the top AI security risks for a reason: four consequences follow from a successful copy.
The examples above show how lopsided the economics are: about $50 in API fees for Mindgard’s Model Leeching clone, under $20 for the Carlini team’s layer extraction, and millions of API exchanges, not years of research, for the distillation campaigns against Claude. The originals cost far more. Per IAPS analysis of Epoch AI data, OpenAI and Anthropic alone have spent more than $18 billion on R&D compute since 2024. An attacker can also sell inference from the clone at a lower price, undercutting the provider on its own product.
A distilled model inherits capability but not safeguards. Anthropic’s February disclosure warns that models built through illicit distillation are “unlikely to retain those safeguards,” meaning the protections that stop a model from helping with tasks like bioweapons development or malicious cyber operations. Its October follow-up gave a concrete case: Zhipu distilled the cyber capabilities of leading US models while building GLM-5.3, which the US government’s CAISSI then assessed as the most cyber-capable open-weights model released to date, and whose own safeguards Anthropic found easy to bypass. The clone is the original with the brakes removed.
A copy gives attackers somewhere to practice without being seen. In Model Leeching, Mindgard’s researchers developed adversarial attacks against their extracted model and then used them against ChatGPT-3.5-Turbo, raising the attack success rate by 11%. Earlier Mindgard research, PINCH, found that even a partially successful extraction makes follow-on adversarial attacks significantly more effective. Google’s threat tracker makes the same point from the defender’s side, warning that a coding model could be targeted by an adversary wanting to “replicate capabilities in an environment without guardrails.” Any model extracted today becomes a test bench for prompt injection, jailbreaks and evasion against the real deployment tomorrow.
A stolen copy can also be probed for what it learned from its training data, an attack known as model inversion, without the original provider ever seeing the queries. Reasoning traces carry a separate risk. When Panfilov et al. decoded reasoning blocks from agent sessions that developers had shared publicly, they recovered 182 credentials, including API keys, passwords and access tokens, along with 367 pieces of personal information.
The stakes show up in how labs talk about their model weights. In a 2023 interview, later quoted in April 2026 testimony to the House Select Committee on the CCP, Anthropic’s CISO said of the weights: “I probably spend almost half of my time as a CISO thinking about protecting that one file.” Extraction is how attackers get much of what’s in that file without ever touching it.
Extraction has a place in every major AI security framework, which makes it easy to scope into a test plan and report upward.
The 2023 list carried a standalone entry, LLM10 Model Theft. The 2025 list folded it into LLM10:2025 Unbounded Consumption, which covers denial of wallet, resource exhaustion and model extraction under one heading. That entry’s attack scenario 5, “Functional Model Replication,” describes using the API to generate synthetic training data and fine-tune another model. The 2026 edition kept the category and moved it up to LLM06:2026.
The OWASP Machine Learning Security Top 10 keeps the older framing as ML05:2023 Model Theft.
ATLAS™ files extraction under the Exfiltration tactic as AML.T0024 Exfiltration via AI Inference API, with sub-techniques for membership inference, inversion and model extraction. The related technique AML.T0005 Create Proxy AI Model describes the attacker’s use of the stolen copy to develop attacks offline. Mindgard maps its attack library to ATLAS™ through the MITRE ATLAS™ Adviser.
NIST AI 100-2 E2025, the adversarial machine learning taxonomy, classifies model extraction as a privacy attack alongside data reconstruction and membership inference. The NIST AI Risk Management Framework places the controls under its Measure and Manage functions.
Tramèr’s 2016 finding still holds: restricting outputs slows extraction but doesn’t stop it. Even when the API returned only a label with no confidence score, the attack still worked; it just took more queries. Google’s 2026 threat tracker offers a more useful design goal. It describes defenses that “can degrade student model performance,” meaning they make the copy the attacker trains (the “student”) worse. The aim is not to make extraction impossible but to make the clone less accurate and the campaign more expensive.
The eight controls below do that, or, in the case of watermarking, help prove theft afterward. Each one is a trade-off against legitimate users, so the table lists the cost.
Three points of practice follow from the documented cases.
Detect the pattern, not the prompt. Anthropic identifies distillation by volume concentrated in a few areas, repetitive prompt structure and content that maps to high-value training data, none of which shows up in a single request. Detection has to run at the account-cluster level, which means logging prompts and metadata with that analysis in mind.
Treat the API surface as an attack surface. Logit bias, logprobs, verbose error messages and encrypted reasoning blocks were each a convenience feature until a paper turned it into an extraction channel. Every new output field needs the question: what does this leak per call?
Measure your own extractability. A red team can run the Model Leeching playbook against your deployment: fix a query budget, train a student, measure agreement and test whether attacks staged on the student transfer. The result is a number your security program can track across releases, the same way continuous AI pentesting tracks jailbreak success rates. Extraction testing belongs in the same AI red teaming scope as prompt injection and data poisoning. For the platforms that run these tests, see our roundup of AI red teaming tools.
Use the checker at the beginning of this post to assess your API deployment and identify the gaps an attacker could exploit first.
Mindgard’s research team wrote the Model Leeching and PINCH papers, and that work is built into the platform’s attack library. Automated AI red teaming runs extraction, inversion and membership inference attacks against your models and agents alongside prompt injection and jailbreak testing, on every build.
Findings map to OWASP Top 10 for LLM Applications 2026 and MITRE ATLAS™ AML.T0024, and confirmed patterns feed runtime detection so the harvesting campaign is caught at the account level, not in next year’s model release. Teams that want a scoped engagement before committing to a platform can start with Mindgard’s AI red teaming services. Book a demo to see what your API gives away.
A model extraction attack is a black-box attack in which an adversary queries a deployed machine learning model through its API, records the outputs and trains a copy that replicates the model’s behavior. The attacker never accesses the weights or training data. The terms model stealing and model theft describe the same attack.
Yes, with a narrower target. A distillation attack is model extraction aimed at a large language model, where the stolen outputs, often chain-of-thought reasoning, are used as training data for a smaller student model. Anthropic and Google both documented industrial-scale distillation attacks against Claude and Gemini in February 2026.
Model extraction steals the model. Model inversion steals the training data by reconstructing it from the model’s outputs. The two often chain: a stolen copy makes inversion against the original easier, which is why MITRE ATLAS™ lists them as sibling sub-techniques under AML.T0024.
It depends on what is being extracted and how much each response reveals. Tramèr et al. extracted simple commercial models in a few hundred to a few thousand queries in 2016, and Mindgard’s Model Leeching attack cloned a ChatGPT task for $50 in API fees. The 2026 distillation campaigns against Claude ran to 16 million exchanges because they targeted general capability rather than one task.
No. Any model that answers queries leaks information about itself, and Tramèr et al. showed in 2016 that hiding confidence scores slows extraction without stopping it. The practical goal is to raise the attacker’s cost and degrade the copy through rate limits, output restriction, account verification and behavioral detection, then to test your own extractability.
Querying an API you have legitimate access to is not itself a crime, which is why providers rely on terms of service, account bans and, in 2026, policy pressure. Anthropic, Google and OpenAI all describe unauthorized distillation as a terms-of-service violation, and the Institute for AI Policy and Strategy has argued for Entity List designations and sanctions against labs that run distillation campaigns.
Mindgard secures AI models, agents, and applications against exploitable threats. Spun out of more than a decade of pioneering AI security research at Lancaster University, Mindgard combines offensive security expertise, AI red teaming, attacker intelligence, and automated protection to help organizations discover, assess, and mitigate risk across AI models, agents, and applications.
We found vulnerabilities in AI from OpenAI, Meta and AWS. See how Mindgard finds and proves the weakness attackers could exploit in yours.
