Vanishing and Exploding Gradients Quiz

Answers and explanations appear as you take the quiz.

1. What mathematical operation in backpropagation is the root cause of vanishing and exploding gradients?
   • Repeatedly multiplying per-layer factors via the chain rule
   • Adding gradients across a single layer
   • Dividing the loss by the batch size
   • Taking the square root of the weights
2. Why can sigmoid or tanh units contribute to vanishing gradients when they are saturated?
   • Their derivatives are always larger than one
   • Their derivatives become small in saturated regions, so repeated chain-rule products can shrink
   • They remove the chain rule from backpropagation
   • They force every weight to zero
3. Which architecture is MOST severely affected by gradient problems over long sequences?
   • A two-layer perceptron
   • A single linear regression
   • A recurrent neural network unrolled over many timesteps
   • A lookup table
4. Seeing the training loss suddenly turn into NaN most likely indicates which problem?
   • Vanishing gradients
   • Perfect convergence
   • Too small a learning rate
   • Exploding gradients
5. How do residual (skip) connections help with vanishing gradients?
   • They give the gradient a direct shortcut path backward, bypassing the shrinking multiplications
   • They remove all nonlinearities
   • They increase the learning rate automatically
   • They delete early layers
6. In a simplified analysis of a recurrent linear transformation repeated over many steps, what matrix quantity is a rough signal of growth versus decay?
   • The matrix trace relative to the batch size
   • The spectral radius relative to one
   • The number of hidden units alone
   • The determinant relative to the learning rate
7. A backpropagated gradient is multiplied by a factor below one at each of many layers. What trend should you expect?
   • It must become exactly one
   • It grows exponentially regardless of the factor
   • The signal can shrink exponentially with depth
   • It stops depending on earlier layers after one step
8. Why can a recurrent network be especially vulnerable over a long sequence?
   • The network has no hidden state
   • The output layer is used only once
   • The sequence is processed without a chain rule
   • The recurrent transformation is applied repeatedly across timesteps, compounding its gradient effect