What is Knowledge Distillation?

Compressing knowledge from a large teacher model into a smaller student model.

A smaller model trained to imitate a larger model's behavior while using less compute at inference.

The latest point in time reflected in a model's training data.

Training approaches that let a model keep learning from new data without forgetting prior knowledge.

Training a model with labeled examples that map inputs to known outputs.

A training and behavior-shaping approach where model outputs are guided by a fixed set of written principles.