Knowledge Distillation Quiz

Answers and explanations appear as you take the quiz.

1. In knowledge distillation, what is the student model trained to do?
   • To train a small student model to imitate a large teacher model
   • To make a model larger and more accurate
   • To label data automatically
   • To encrypt a model's weights
2. What is meant by the teacher's 'dark knowledge'?
   • Hidden malicious behavior in the model
   • The encrypted portion of the model file
   • The information in the relative probabilities the teacher assigns to wrong classes
   • Data the teacher was never trained on
3. What role does temperature (T) play in distillation?
   • It controls the model's training speed only
   • It softens the softmax so small inter-class similarities become learnable
   • It sets the number of layers in the student
   • It measures the GPU's heat output
4. The classic distillation loss combines which two components?
   • Two separate reward models
   • Mean squared error and a hinge loss
   • A KL penalty against a reference policy and a value function
   • A KL-divergence term matching the teacher's soft outputs plus cross-entropy on the true labels
5. Which is an example of a distilled language model?
   • ResNet-50
   • Word2Vec
   • DistilBERT
   • AlexNet
6. How does feature-based distillation differ from the classic output-based approach?
   • It matches intermediate hidden-layer representations, not just final outputs
   • It ignores the teacher entirely
   • It only uses hard labels
   • It trains the teacher instead of the student
7. A teacher assigns similar small probabilities to wolf and fox when predicting dog. What can soft-target distillation transfer beyond the top label?
   • A guarantee that either alternative is the correct label
   • The teacher’s complete original training corpus
   • The teacher’s exact number of hidden layers
   • Relative relationships among the teacher’s alternative class scores
8. A distilled student is smaller than its teacher. Which evaluation is needed before deployment?
   • Assume its accuracy must equal the teacher’s
   • Measure its task accuracy and runtime on the intended system
   • Assume fewer parameters guarantee lower latency on every device
   • Only compare the models’ names