引導式測驗 · 中等 等級

梯度檢查點測驗

解釋激活重新實現、記憶體節省、與計劃相關的計算開銷以及與分片和混合精度的組合。

相關引導路徑梯度檢查點
問題 1 的 8

為了節省內存,梯度檢查點主要交換什麼?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. 為了節省內存,梯度檢查點主要交換什麼?
    • 額外計算(重新計算)以減少記憶體使用
    • 速度模型精度
    • 網路頻寬的磁碟空間
    • 大量精度
  2. 為什麼激活通常在前向傳遞過程中儲存?
    • 將它們顯示給用戶
    • 因為反向傳播需要它們來計算梯度
    • 加快資料載入速度
    • 它們從來都不是真正需要的
  3. In a simplified chain of N similarly sized layers, what can checkpoint boundaries spaced about sqrt(N) layers apart achieve in the textbook schedule?
    • It stores no activations and performs no recomputation
    • It guarantees constant memory for every model graph
    • Activation storage can scale on the order of sqrt(N) checkpoints
    • It makes optimizer-state memory scale as sqrt(N)
  4. What most determines gradient-checkpointing runtime overhead in a practical model?
    • Only the model’s file extension
    • A fixed 20–30% slowdown for every model
    • The number of labels in the dataset
    • Which operations are recomputed, checkpoint placement, graph structure, and hardware
  5. 在 PyTorch 中,哪個實用程式通常用於將梯度檢查點應用於模組?
    • torch.utils.checkpoint
    • 火炬優化SGD
    • torch.nn.Dropout
    • torch.cuda.empty_cache
  6. What happens to operations inside a checkpointed region during backward?
    • Their gradients are skipped entirely
    • Their forward computations are rerun to regenerate needed activations
    • Their weights are permanently discarded
    • They are executed only during inference
  7. Which training techniques can be combined with activation checkpointing to reduce different memory costs?
    • Only increasing the number of labels
    • Turning off backpropagation
    • FSDP for sharded model state and mixed precision for lower-precision tensors
    • Replacing the model with a tokenizer
  8. What is selective activation checkpointing intended to choose?
    • Which training labels are removed
    • Which model outputs are published
    • Which GPU becomes the tokenizer
    • Which operations to save and which to recompute based on their memory and runtime costs