Collective Communication and NCCL Quiz

Answers and explanations appear as you take the quiz.

1. What does an all-reduce operation accomplish in distributed training?
   • It combines values across all GPUs and gives every GPU the combined result
   • It deletes redundant data from one GPU
   • It reduces the GPU clock speed to save power
   • It sends data from all GPUs to a single storage disk
2. What does the acronym NCCL stand for?
   • Neural Compute Cluster Layer
   • Network Cache Control Logic
   • NVIDIA Collective Communications Library
   • Numeric Convolution Compute Library
3. Why can chunked ring communication be useful for a large tensor?
   • It guarantees constant latency for any number of devices
   • Chunks can be streamed across participating links instead of waiting for one monolithic transfer
   • It makes topology irrelevant
   • It prevents all network contention
4. Which collective operation gathers each GPU's piece of data into a full copy held by all GPUs?
   • Broadcast
   • Reduce-scatter
   • Barrier
   • All-gather
5. What does a supported network reduction offload such as SHARP do?
   • Guarantee exactly half the traffic for every collective
   • Remove the need for participating ranks
   • Perform some reduction work in compatible network hardware
   • Turn all-gather into a local memory copy
6. Which interconnect does NCCL typically use for communication BETWEEN servers (nodes)?
   • InfiniBand or RoCE
   • A USB cable
   • The PCIe slot only
   • Wi-Fi
7. Three ranks contribute scalar values 2, 3 and 5 to a sum all-reduce. What value does each receive?
   • Only its own original value
   • 3 for the middle rank and nothing elsewhere
   • The average 3.33 unless requested separately
   • 10
8. Which collective combines contributions but distributes reduced pieces instead of the entire reduced buffer to every rank?
   • All-gather
   • Reduce-scatter
   • Broadcast
   • A standalone model checkpoint