Second-Order Optimization and Newton Methods Quiz

Answers and explanations appear as you take the quiz.

1. What information does Newton's method use that plain gradient descent does not?
   • Curvature, via the second derivatives in the Hessian matrix
   • The exact location of the global minimum
   • The learning rate schedule
   • The number of training examples
2. For a perfectly quadratic objective, how many steps does Newton's method need to reach the minimum?
   • Exactly ten steps
   • One step
   • The same number as gradient descent
   • It never converges
3. Why is full Newton's method impractical for billion-parameter neural networks?
   • The gradient cannot be computed
   • Second derivatives are always zero in neural networks
   • The Hessian is N-by-N, costing about N-squared memory and N-cubed compute to invert
   • It only works on images
4. What do quasi-Newton methods like BFGS do to avoid the Hessian's cost?
   • They ignore the gradient entirely
   • They randomly guess the curvature
   • They convert the problem to a linear one
   • They build an approximation of the inverse Hessian from successive gradient differences
5. How does L-BFGS reduce memory compared to BFGS?
   • It stores only the last few gradient and step vectors instead of the full matrix
   • It uses single-precision floats only
   • It discards the gradient after each step
   • It trains on fewer parameters
6. Which modern approach approximates curvature to speed up large neural network training?
   • One-hot encoding
   • Kronecker-factored methods like K-FAC and Shampoo
   • Plain stochastic gradient descent with no modifications
   • Dropout
7. On an exact quadratic objective with an invertible Hessian, what assumption supports the one-step Newton result?
   • The learning rate is reduced to zero before the update
   • The Hessian is ignored and only a fixed scalar step is used
   • The full Newton update uses the exact gradient and Hessian of that quadratic
   • The objective is first converted to a linear function
8. A non-quadratic objective is optimized with Newton’s method. What role does its quadratic model play?
   • It guarantees the global minimum in one step
   • It replaces the need to evaluate a gradient
   • It makes every Hessian positive definite
   • It approximates local curvature, so later iterations may be needed as the model changes