Up nextNext guide
Adversarial Diffusion Distillation and Turbo Models
Technical
Technical GUIDE
Knowledge Distillation
Knowledge distillation trains a student model using a teacher model’s outputs or representations as supervision.
Listen to this guideAI-narrated · ~2 min
On this page2 min read
Overview
It can support more efficient deployment, but the student’s accuracy and behavior depend on its capacity, training data, and the teacher signal.
Deep Dive
Big models are accurate but slow and expensive to deploy. Knowledge distillation transfers their capability into a compact model by having the student learn from the teacher's outputs rather than only from hard labels. The key insight, from Hinton and colleagues, is that a teacher's full probability distribution carries 'dark knowledge': even when it predicts 'dog', the relative probabilities for 'wolf' versus 'car' reveal how the teacher sees similarities. Softening these probabilities with a temperature exposes that structure, and the student is trained to match it, often alongside the true labels. The goal is an efficient student that benefits from teacher supervision; the actual accuracy and speed must be measured for the task and implementation. DistilBERT and TinyBERT are well-known distilled language models.
Real-World Implementation
DistilBERT compressing BERT to roughly 40% fewer parameters while retaining most of its language understanding for faster inference.
Shrinking a large vision model so an image classifier can run in real time on a smartphone camera app.
Distilling a big model's chain-of-thought reasoning into a smaller model to make it answer math or coding questions more cheaply.
Compressing an ensemble of models into a single student so production serving costs and latency drop without much accuracy loss.
The Future of Knowledge Distillation
Distillation is now a standard step in shipping efficient models and is central to today's wave of small, capable open models. A fast-growing trend is sequence-level distillation from large language models, where a strong model generates training data or reasoning traces (including chain-of-thought) to teach smaller students, blurring the line with synthetic data. Expect tighter pairing with quantization and pruning, more on-device deployment, and ongoing debate about licensing and quality when distilling from proprietary models whose outputs become a competitor's training signal.
Putting it into practice
Why it matters
- Cost and budget. Architecture decisions drive performance and operating cost for years.
- Clearer decisions. Technical education helps teams choose the right stack, not just the newest one.
- Quality control. Better engineering choices reduce reliability incidents in production.
Watch out for
- Optimizing one benchmark can hide broader system weaknesses.
- Infrastructure and maintenance costs are often underestimated.
- Security and observability gaps can grow as systems become more complex.
Next steps
- Define latency, quality, and cost targets before implementation.
- Benchmark under realistic load and data conditions.
- Instrument monitoring for errors, drift, and user impact.
- Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Knowledge Distillation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is Knowledge Distillation?
Knowledge distillation trains a student model using a teacher model’s outputs or representations as supervision. It can support more efficient deployment, but the student’s accuracy and behavior depend on its capacity, training data, and the teacher signal.
In knowledge distillation, what is the student model trained to do?
Distillation transfers a big teacher's capability into a compact, faster student model.
What is meant by the teacher's 'dark knowledge'?
Dark knowledge is the structure in the teacher's full probability distribution, like how similar 'wolf' is to 'dog', not just the top answer.
What role does temperature (T) play in distillation?
A higher temperature flattens the probability distribution, revealing the relative similarities the student should learn.
The classic distillation loss combines which two components?
The student matches the teacher's softened distribution (KL term) while also fitting the ground-truth labels (cross-entropy).
Which is an example of a distilled language model?
DistilBERT is a well-known model distilled from BERT, retaining most performance with far fewer parameters.
Keep learning
Related guides
More guides picked for this topic