Optimizer State Offloading to CPU and NVMe Quiz

Answers and explanations appear as you take the quiz.

1. What is the primary goal of offloading optimizer states to CPU or NVMe?
   • To make the model more accurate
   • To reduce the number of training epochs needed
   • To eliminate the need for gradients entirely
   • To fit larger models by freeing up scarce GPU memory
2. For an Adam setup with two FP32 moments and an FP32 master weight, how much memory do those three values use per parameter?
   • Two bytes for all three values
   • Twelve bytes, excluding model weights and gradients
   • Sixteen bytes for the three values alone
   • Zero bytes because they are implicit
3. Which framework feature popularized large-scale optimizer offloading?
   • A new activation function
   • JPEG image compression
   • DeepSpeed ZeRO-Offload and ZeRO-Infinity
   • Gradient clipping
4. What is the main performance cost of offloading to CPU or NVMe?
   • Data must travel over slower links like PCIe or to disk, adding latency
   • The model forgets earlier training
   • The optimizer becomes mathematically incorrect
   • GPU temperature drops too low
5. How do offloading systems hide much of the transfer latency?
   • By skipping the backward pass
   • By overlapping data prefetching with ongoing GPU computation
   • By training only one parameter at a time
   • By disabling mixed precision
6. In ZeRO-Offload, where does the Adam optimizer step often execute?
   • Entirely on the NVMe controller
   • On a separate display GPU
   • On the CPU, using the master weights and moments stored there
   • Inside the network switch
7. Optimizer states are offloaded, but GPU weights and activations still exceed capacity. What follows?
   • The job can still run out of GPU memory
   • Offloading guarantees the job fits any GPU
   • The optimizer moments automatically become model activations
   • The model no longer needs forward tensors
8. A CPU-offloaded optimizer becomes the training bottleneck. Which measurement is most relevant?
   • Only the inference accuracy
   • The UI font used for training logs
   • Only the number of dataset classes
   • Host optimizer-update time and host-device transfer time