Fully Sharded Data Parallel Quiz

Answers and explanations appear as you take the quiz.

1. What does FSDP shard across GPUs that standard data parallelism does NOT?
   • Parameters, gradients, and optimizer states
   • Only the input batches
   • Only the learning rate schedule
   • The training dataset files on disk
2. Which collective operation does FSDP use to reconstruct a layer's full weights right before computing it?
   • Broadcast-only
   • All-gather
   • Scatter
   • Barrier
3. Why does FSDP free the gathered full weights immediately after a layer's computation?
   • To corrupt unused gradients
   • To force a slower training loop
   • To keep the permanent memory footprint to just one shard per device
   • Because PyTorch deletes them randomly
4. FSDP was largely inspired by which earlier memory-optimization approach?
   • Google's MapReduce
   • Apache Spark RDDs
   • NVIDIA's CUDA cores
   • Microsoft's ZeRO (Zero Redundancy Optimizer)
5. How does FSDP hide much of the network latency from gathering weights?
   • By overlapping communication with computation, prefetching the next layer's parameters
   • By disabling the backward pass
   • By compressing weights to one bit
   • By training on CPU only
6. Which collective operation distributes combined gradient slices back to their owning GPUs in the backward pass?
   • All-to-all duplication
   • Reduce-scatter
   • Single-GPU copy
   • Gather-only
7. Why can fully sharding state lower persistent per-GPU memory compared with replicated data parallelism?
   • It removes all model parameters during training
   • It converts optimizer states into network requests
   • Each rank stores a shard of model state instead of a full copy
   • It places every activation permanently on the CPU
8. What is a central systems tradeoff in FSDP?
   • Lower memory and no communication at any point
   • More replicated parameters in exchange for fewer data samples
   • Replacing training with inference
   • Lower per-device memory use in exchange for collective communication