Stochastic Weight Averaging Quiz

Answers and explanations appear as you take the quiz.

1. What does Stochastic Weight Averaging actually average?
   • The weights of the model sampled at multiple points late in training
   • The predictions of several separately trained models
   • The gradients computed across all mini-batches
   • The learning rates used during each epoch
2. Why does SWA tend to improve generalization?
   • It memorizes the training set more precisely
   • It finds wider, flatter minima that are more robust on unseen data
   • It increases the number of parameters in the model
   • It eliminates the need for a validation set
3. What extra step does SWA require for networks that use batch normalization?
   • Retraining the network from scratch
   • Doubling the batch size
   • An additional forward pass to recompute BN running statistics
   • Removing all dropout layers
4. What learning-rate behavior is typically used during the SWA averaging phase?
   • A learning rate decayed to exactly zero
   • A learning rate of one for stability
   • No learning rate at all since training is frozen
   • A constant or cyclical, relatively high learning rate
5. How does SWA's inference cost compare to a traditional ensemble of N models?
   • It costs roughly a single model, since the result is one set of weights
   • It is N times more expensive
   • It is identical to running N models
   • It cannot be used for inference
6. Why does the batch-normalization adjustment belong after SWA weight averaging?
   • The averaging step deletes every convolution
   • Its running statistics must be recomputed for the averaged weights
   • It converts batch normalization into layer normalization
   • It restores the original optimizer learning rate
7. An SWA running mean combines n previous samples with a new weight vector w_i. Which update gives equal weight to all n+1 samples?
   • w_SWA + n · w_i
   • (w_SWA + w_i) divided by n
   • (n · w_SWA + w_i) divided by (n + 1)
   • n · (w_SWA − w_i)
8. Why is SWA commonly started during the later part of training?
   • It requires the model to remain at random initialization
   • It can only average models with different architectures
   • It replaces the training data with validation examples
   • It averages nearby useful checkpoints from a trajectory after the model has made substantial progress