Probability Calibration Quiz

Answers and explanations appear as you take the quiz.

1. A model is well calibrated if, among predictions it labels 80% confident, the event occurs:
   • About 80% of the time
   • 100% of the time
   • About 20% of the time
   • Exactly once
2. What does temperature scaling do to a neural network's outputs?
   • Changes which class is predicted
   • Divides the logits by a learned scalar before softmax, adjusting confidence
   • Adds noise to the training labels
   • Retrains the network from scratch
3. What does a reliability diagram plot?
   • Training loss versus epoch
   • Model size versus accuracy
   • Predicted probability versus observed frequency
   • Learning rate versus batch size
4. Expected Calibration Error (ECE) measures:
   • The model's top-1 accuracy
   • The number of parameters in the model
   • The variance of the gradients
   • The weighted average gap between confidence and accuracy across bins
5. Modern deep neural networks are most often described as:
   • Overconfident
   • Underconfident
   • Perfectly calibrated by default
   • Unable to output probabilities
6. Why can a highly accurate model still need calibration?
   • Accurate models always have zero calibration error
   • Accuracy and calibration are distinct properties
   • Calibration only matters for regression
   • Accuracy guarantees honest probabilities
7. On which data is temperature scaling typically fit in the guide’s calibration workflow?
   • The final test set using classification accuracy
   • Only the training labels without model outputs
   • A held-out validation set using negative log-likelihood
   • The model’s hidden weights using mean squared error
8. Why can temperature scaling improve calibration without changing top-1 accuracy?
   • It changes the model’s predicted class labels randomly
   • It retrains the feature extractor
   • It replaces every logit with the same value
   • Dividing logits by a positive scalar changes confidence but preserves their ordering