AI Glossary Term

What does “Knowledge Distillation” mean?

Definition

Training a smaller model to imitate the outputs of a larger model.

Browse the full AI Glossary Explore all free AI guides

How Knowledge Distillation works

Example

A large image classifier says an image is 80% cat, 15% lynx, and 5% dog. A small student model trained on those soft scores learns that cats and lynxes look alike, which carries more information than the single label 'cat'.

Why it matters

Distillation yields smaller models that are cheaper and faster to run. It is a common way to deploy AI on limited hardware.

Common questions

How does it relate to the term Distillation?

They mean essentially the same technique; knowledge distillation is the fuller name.

Who proposed it?

Geoffrey Hinton and colleagues popularized the approach in a 2015 paper.

Is the student always smaller?

Typically, but the idea can also be used between similarly sized models to improve training.

DistillationCompressing knowledge from a large teacher model into a smaller student model.Distilled ModelA smaller model trained to imitate a larger model's behavior while using less compute at inference.Knowledge CutoffThe latest point in time reflected in a model's training data.Continual LearningTraining approaches that let a model keep learning from new data without forgetting prior knowledge.Supervised LearningTraining a model with labeled examples that map inputs to known outputs.Constitutional AIA training and behavior-shaping approach where model outputs are guided by a fixed set of written principles.

Learn more in our free guides

AI Knowledge Management DINO Self-Distillation GraphRAG Knowledge Graphs DreamFusion and Score Distillation Sampling

Test yourself

AI Knowledge Management Quiz DINO Self-Distillation Quiz GraphRAG Knowledge Graphs Quiz