Society GUIDE
Model Extraction and Stealing Attacks
Model extraction uses query access to a deployed model to train a substitute that imitates some of its behavior.
Listen to this guideAI-narrated · ~2 min
On this page2 min read
Overview
Attacks and reported recovery are specific to the model interface, outputs, and experiment.
Deep Dive
A model-extraction attack treats a deployed model as an oracle: an attacker collects input-output pairs and trains a substitute. The 2016 prediction-API paper demonstrates extraction attacks for specific model classes and shows that omitting confidence values does not prevent every attack. A 2024 study, *Stealing Part of a Production Language Model*, recovers an embedding-projection layer up to symmetries: it reports exact projection-matrix extraction for Ada and Babbage for under $20 in queries and estimates under $2,000 for GPT-3.5 Turbo. These results concern that layer and those models, not full checkpoint extraction. DynaMarks is a separate proposal that dynamically changes API responses using secret parameters to watermark surrogate models in its evaluated vision-model settings.
Real-World Implementation
Limit and monitor high-volume queries to a prediction API.
Compare label-only responses with probability-vector responses when reviewing information exposure.
Evaluate a proposed extraction defense on the model class and access pattern it targets.
Treat watermarking as a method-specific defense, not a universal guarantee of ownership.
The Future of Model Extraction and Stealing Attacks
Defenses may combine access controls, monitoring, response design, and model-specific watermarking research. Assess each mitigation against the attack setting and model class it was tested on; no single safeguard guarantees that a substitute cannot be trained or identified.
Putting it into practice
Why it matters
- Risk and safety. Catastrophic and everyday AI harms both depend on who understands the risks and who can act.
- Clearer decisions. Public and professional literacy shapes whether strong safety policy is politically possible.
- Cutting through hype. Clear explanations reduce capture by hype, lab PR, and vague ethics theater.
Watch out for
- Treating existential risk as sci-fi while capability compounds.
- Confusing surface product safety with alignment under high autonomy.
- Leaving non-English and non-expert audiences with only low-quality sources.
Next steps
- Separate product harms, misuse, and loss-of-control / misalignment risks.
- Ask what evidence would change your view on timelines and severity.
- Prefer primary sources and concrete evals over marketing claims.
- Identify one action path: career, policy, funding, or skills — not only awareness.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Model Extraction and Stealing Attacks quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is Model Extraction and Stealing Attacks?
Model extraction uses query access to a deployed model to train a substitute that imitates some of its behavior. Attacks and reported recovery are specific to the model interface, outputs, and experiment.
What is the core idea behind a model extraction attack?
Extraction treats the deployed model as an oracle: the attacker collects input-output pairs and trains a copycat model to mimic the behavior, without ever accessing the original weights.
Why does returning full probability vectors make extraction easier than returning only a top-1 label?
Each full probability vector reveals far more about the model's internal decision surface than a single label, letting the attacker reconstruct the boundary with fewer queries.
Which defensive technique directly reduces information leaked per query?
Coarsening outputs, for example returning only the top label or rounded probabilities, reduces the signal each response gives an attacker, raising the cost of extraction.
What did the 2024 production-language-model extraction paper report about GPT-3.5 Turbo?
The paper recovers the embedding-projection layer up to symmetries, exactly extracts the projection matrix for Ada and Babbage for under $20, and estimates under $2,000 in queries for GPT-3.5 Turbo.
Why is a stolen substitute model dangerous beyond just lost revenue?
Once attackers have a local copy, they can probe it freely to design adversarial inputs or evasion attacks that also fool the original deployed system.
Keep learning
Related guides
More guides picked for this topic