引導式測驗 · 中等 等級
啟動重新計算權衡測驗
測試啟動檢查點記憶體權衡、選擇性重新計算和重播正確性,包括為什麼額外的計算不能解決所有記憶體成本。
問題 1 的 8
啟動重新計算會犧牲什麼來節省記憶體?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- 啟動重新計算會犧牲什麼來節省記憶體?
- 模型精度
- 額外計算(FLOPs)
- 網路頻寬
- 數值精度
- 為什麼前向傳遞激活通常會被儲存?
- 將它們顯示給用戶
- 壓縮模型
- 它們是標記化所必需的
- 因為反向傳播需要它們來計算梯度
- 為什麼應該在實際模型上測量 activation checkpointing 的開銷?
- Checkpoint placement, operators, and hardware determine the cost of replay
- It always adds exactly forty percent to wall-clock time
- It guarantees a tenfold speed improvement
- It removes all forward computation
- 對於理想化的 N 層鏈且激活相似,平方根檢查點放置能實現什麼樣的儲存縮放?
- O(N squared) for all model parameters
- O(1) for all training memory regardless of N
- O(sqrt(N)) for activation storage
- O(N squared) for optimizer state
- 選擇性(非完整)重新計算背後的想法是什麼?
- 每個操作重新計算兩次
- 永遠不要重新計算任何東西
- 僅重新計算嵌入層
- 僅重新計算佔用大量記憶體但計算成本低的操作,快取昂貴的操作
- 哪種補充技術經常與重新計算相結合以節省更多記憶體?
- 增加批量大小
- 將啟動卸載到 CPU 或 NVMe
- 刪除所有標準化層
- 切換到推理模式
- A checkpointed block contains dropout. Why does consistent RNG state matter during replay?
- Dropout becomes a model-weight quantizer
- The optimizer no longer needs gradients
- Backward recomputation should reproduce the intended forward randomness
- The batch size must become one
- Checkpointing reduces stored activations, but optimizer states still exceed GPU memory. Why is the job still out of memory?
- Activation recomputation does not remove optimizer-state storage
- Checkpointing increases every optimizer tensor to FP64
- Backward no longer consumes any memory
- The model cannot use checkpointing with an optimizer