技術指南

啟動重新計算權衡

激活重算,也稱為激活或梯度檢查點,透過丟棄前向傳遞的中間結果並在反向傳遞需要時重新計算,節省神經網路訓練時的 GPU 記憶體。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 估算一層 GPT-3 大小的激活記憶體。
  5. 現實世界的實施
  6. 啟動重新計算權衡的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

通常在每層應用時會增加 30% 至 40% 的計算量,選擇重算哪些層是訓練大型模型的核心杠杆。

重點摘要

  1. 激活重算只儲存部分前向傳遞的激活,並在反向傳遞時重新計算其餘部分,換取額外計算以降低記憶體使用。
  2. Chen 等人在 2016 年證明,每隔 n 層的平方根層進行檢查點可將激活記憶體降低到 n 的平方根次序。
  3. 完整重算在 NVIDIA 於 2022 年對大型變壓器模型的測量中增加了 30% 至 40% 的執行時間。
  4. 對注意力操作的選擇性重算在序列並行下將激活記憶體降低約五倍,並恢復約 30% 的吞吐量。
  5. 重算必須精確重現前向傳遞,因此 dropout 的隨機狀態必須恢復,非決定性程式碼可能破壞梯度。

深入探討

Training a neural network with backpropagation has two phases. In the forward pass, the input flows through each layer and every layer produces an output, called an activation. In the backward pass, the training algorithm works from the loss back toward the input, computing a gradient for every weight. Most of those gradient calculations need the activations from the forward pass, so by default a framework such as PyTorch or JAX keeps every activation in memory until the backward pass uses it. Activation recomputation, also known as activation checkpointing, gradient checkpointing or rematerialization, changes that rule: it keeps only some activations, discards the rest, and recomputes the discarded ones from the nearest kept value when the backward pass needs them.

The reason this matters is that activation memory, unlike weight memory, grows with the size of each training batch and with sequence length. For large transformers it often dominates. Korthikanti and colleagues at NVIDIA give a formula for a single transformer layer with no model parallelism: activation memory in bytes is roughly s times b times h times (34 + 5as/h), where s is sequence length, b is microbatch size, h is hidden size and a is the number of attention heads. The 5as/h part comes from attention, which builds a score for every pair of tokens, so it grows with the square of the sequence length. For a GPT-3-sized configuration that term equals 80, larger than the 34 from everything else. Multiplied across nearly a hundred layers, the stored activations can exceed the memory of any single GPU even before the weights are counted.

The mechanism is simple to state. Pick a set of checkpoints, usually the inputs to certain layers or blocks. During the forward pass, save only those checkpoints and throw away everything computed between them. During the backward pass, when gradients reach a block, rerun that block's forward computation from its saved input to rebuild its internal activations, use them to compute gradients, and free them again before moving to the previous block. Only one block's worth of full activations needs to be in memory at a time, plus the checkpoints. The price is that most forward computation is done twice.

The idea was brought into mainstream deep learning practice by Tianqi Chen, Bing Xu, Chiyuan Zhang and Carlos Guestrin in the April 2016 paper Training Deep Nets with Sublinear Memory Cost. They showed that for a network of n layers, placing a checkpoint every square root of n layers cuts activation memory from proportional to n to proportional to the square root of n, at the cost of roughly one extra forward pass per mini-batch. They also showed a more extreme version that reduces memory to proportional to log n at the cost of order n log n extra forward computation. Their headline example reduced the memory for a 1,000-layer residual network from 48 GB to 7 GB with about 30 percent more running time. In 2019, the Checkmate paper by Paras Jain and colleagues at Berkeley, published at MLSys 2020, framed checkpoint selection as a mixed integer linear program that finds an optimal recomputation schedule for a given memory budget in under an hour with off-the-shelf solvers, and reported training with up to 5.1 times larger inputs.

For large language models, the key refinement is selective recomputation. Korthikanti and colleagues observed in May 2022 that not all activations are equally expensive to recompute. The attention operations, meaning the query-key product, the softmax, softmax dropout and the weighted sum over values, create large activations but require relatively few floating point operations per element. The large matrix multiplications in the feed-forward layers are the opposite. By recomputing only the attention pieces and storing everything else, combined with sequence parallelism, which splits certain layer operations across GPUs along the sequence dimension, they reduced activation memory by about five times and cut the time overhead of recomputation by more than 90 percent. For a single 22-billion-parameter layer, they measured full recomputation adding 39 percent execution time and the combined approach only 4 percent. End to end, training throughput rose by about 30 percent across models from 22 billion to 1 trillion parameters.

Recomputation competes with several alternatives, and real systems mix them. Doing nothing is fastest when memory allows. Shrinking the microbatch reduces activation memory linearly but hurts GPU efficiency. Offloading moves activations to CPU memory and back, trading compute for data transfer; PyTorch's selective checkpointing API and JAX's policies both include offload options. Model parallelism splits activations across GPUs at the cost of communication. Fused kernels such as FlashAttention, posted in May 2022 by Tri Dao and colleagues, never write the full attention matrix to GPU memory: they store the softmax normalization statistics and recompute attention blocks on-chip during the backward pass. The paper reports this makes memory linear in sequence length and speeds up the backward pass despite the extra arithmetic, because reading and writing GPU high-bandwidth memory was the real bottleneck.

The limitations are mostly about cost and correctness. Full recomputation adds roughly 30 to 40 percent of execution time in the NVIDIA measurements, which translates directly into training cost. Placing checkpoints poorly saves little memory: the JAX documentation notes that checkpointing the entire differentiated function, or only its last stage, saves nothing and wastes compute. Recomputation must also reproduce the forward pass exactly. Layers that use randomness, such as dropout, need the random number generator state restored, which PyTorch does by default through preserve_rng_state at some performance cost. The PyTorch documentation warns that if a checkpointed function behaves differently on the second run, for example because it reads a global variable, the result can be an error or silently incorrect gradients.

In practice, using recomputation requires little code but careful measurement. In PyTorch, wrapping each transformer block with torch.utils.checkpoint.checkpoint and passing use_reentrant=False, which the documentation recommends, is the usual starting point; selective policies can then save specific operations such as matrix multiplies. In JAX, jax.checkpoint with a policy does the same, and the documentation notes that under jax.jit the XLA compiler often makes these choices automatically. The right setting is found by profiling peak memory and step time together: the goal is the largest batch or sequence length that fits, at the lowest recomputation overhead, since every percent of overhead is paid on every training step.

04工作範例

估算一層 GPT-3 大小的激活記憶體。

  1. 考慮一個設定,序列長度 s = 2,048,微批次 b = 1,隱藏層大小 h = 12,288,注意力頭數 a = 96。

  2. 計算 sbh = 2,048 x 1 x 12,288 = 25,165,824,以及注意力項 5as/h = 5 x 96 x 2,048 / 12,288 = 80。

  3. 如果不進行重計算,每層記憶體為 sbh x (34 + 80) = 約 28.7 億位元組,即大約 2.9 GB。

  4. 如果對注意力進行選擇性重計算,注意力項將消失:sbh x 34 = 每層約 0.86 GB。

  5. 如果進行完全重計算,僅保留層輸入:2sbh = 每層約 50 MB。

  6. 將每個數字乘以 96 層以比較總量,分別約為 275 GB、82 GB 和 4.8 GB。

它顯示了什麼

學習者將明白為什麼選擇性重計算通常是中間方案:它消除了約 70% 的激活記憶體,與 NVIDIA 報告的 GPT-3 的減少量相符,而完全重計算雖然節省了更多記憶體,但幾乎重複了整個前向傳播過程。這些計算可以根據 2022 年 NVIDIA 論文中的公式進行核對。

現實世界的實施

Chen 及其同事在 2016 年報告說,重計算將訓練 ImageNet 上 1,000 層殘差網路所需的記憶體從 48 GB 減少到 7 GB,代價是運行時間增加約 30%。

NVIDIA 研究人員使用選擇性重計算與序列並行,在 2,240 個 A100 GPU 上訓練了一個 5300 億參數的 GPT-3 風格模型,將模型 FLOPs 利用率從 42.1 提高到 54.2 百分比。

PyTorch 將此技術公開為 torch.utils.checkpoint,透過包裝諸如 transformer 層之類的區塊,使得 autograd 僅儲存其輸入並在反向傳播期間重新執行該區塊。

JAX 提供具有命名策略的 jax.checkpoint(例如 dots_with_no_batch_dims_saveable),可保留昂貴的矩陣相乘結果並重新計算較便宜的逐元素運算。

FlashAttention(2022)避免儲存完整的注意力矩陣,並在反向傳播時於晶片上重新計算,作者報告稱這使得反向傳播更快,而非更慢,因為它減少了記憶體流量。

啟動重新計算權衡的未來

Recent work treats recomputation as one decision in a joint plan with parallelism, offloading and kernel fusion rather than a switch. Checkmate (2019) showed optimal schedules can be solved for a memory budget, and the 2022 NVIDIA work showed that choosing operations by memory per FLOP recovers most of the lost throughput. Framework APIs have followed: PyTorch's documentation describes selective activation checkpointing with per-operation save, recompute and CPU offload policies, and says that omitting the use_reentrant argument raises an exception starting in version 2.9, pushing users to the newer non-reentrant implementation. JAX documents named policies that save matrix multiplies or offload them to host memory. Open problems include automatic policies that adapt to sequence length during training, interactions with compilers that already rematerialize values on their own, and long-context training, where the attention term that grows with the square of sequence length makes the choice of what to store more important. FlashAttention's 2022 result, that recomputation can be faster than storage when memory bandwidth is the bottleneck, suggests that the cost model for these decisions depends on hardware and will keep shifting as GPU memory and bandwidth change.

付諸實踐

為什麼這很重要

  • 成本與預算. 多年來,架構決策決定著效能和營運成本。
  • 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
  • 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。

留意

  • 優化一項基準測試可以隱藏更廣泛的系統弱點。
  • 基礎設施和維護成本常常被低估。
  • 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。

後續步驟

  1. 在實施之前定義延遲、品質和成本目標。
  2. 在實際負載和資料條件下進行基準測試。
  3. 儀器監控錯誤、漂移和使用者影響。
  4. 在擴展之前準備回滾和事件回應路徑。

資料來源與延伸閱讀

  1. arXiv (Chen, Xu, Zhang, Guestrin)以次線性記憶體成本訓練深度神經網路
  2. arXiv (NVIDIA)減少大型變壓器模型中的啟動重新計算
  3. arXiv / MLSys 2020Checkmate:以最佳張量再現化突破記憶體牆
  4. arXiv (Dao et al.)FlashAttention:具 IO 感知的快速且記憶體高效的精確注意力
  5. PyTorch documentationtorch.utils.checkpoint
  6. JAX documentation使用 jax.checkpoint(jax.remat)進行梯度檢查點

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

進行 Activation Recomputation Tradeoffs 測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是 Activation Recomputation Tradeoffs?

Activation recomputation,也稱為 activation 或 gradient checkpointing,透過在前向傳遞時丟棄中間結果並在反向傳遞需要時重新計算,來節省神經網路訓練中的 GPU 記憶體。將其應用於每一層通常會增加 30% 至 40% 的計算量,而選擇哪些層重新計算是訓練大型模型的關鍵杠杆。

啟動重新計算會犧牲什麼來節省記憶體?

重新計算會丟棄儲存的啟動並在向後傳遞中重新產生它們,從而花費額外的計算來減少記憶體使用。

為什麼前向傳遞激活通常會被儲存?

向後傳遞使用前向激活來計算梯度,因此預設情況下它們會保留在記憶體中,直到向後傳遞運行。

為什麼應該在實際模型上測量 activation checkpointing 的開銷?

重新計算會消耗額外的計算資源,但實際執行時間取決於所選操作和系統。

對於理想化的 N 層鏈且激活相似,平方根檢查點放置能實現什麼樣的儲存縮放?

理論上的平方根結果僅適用於適當的鏈模型中的 activation checkpointing,而非所有訓練記憶體類別。

選擇性(非完整)重新計算背後的想法是什麼?

選擇性重新計算的目標是使用大量記憶體但計算量很少的操作(例如 Softmax 或 Layernorm),同時快取昂貴的 GEMM 結果以最大程度地減少浪費的 FLOP。