概述
It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.
深入探討
Traditional training stores weights and runs math in 32-bit floating point (FP32). Mixed precision uses lower-precision FP16 or bfloat16 for eligible operations while retaining FP32 where needed. Some FP16 recipes maintain a separate FP32 master copy to accumulate small weight updates; autocast workflows may instead keep model parameters in FP32 while casting selected operation inputs. Using 16-bit tensors can reduce memory used by eligible operations. Supported accelerators may execute some low-precision operations faster, but end-to-end throughput depends on hardware, workload and which operations autocast selects. FP16 has a narrower range, so tiny gradients can underflow. Loss scaling multiplies the loss before backpropagation and reverses that scale before the optimizer update; tools such as torch.amp.GradScaler can manage this process. Bfloat16 has a wider exponent range than FP16 and often does not need the same loss-scaling strategy, though model and hardware behavior still need testing.
現實世界的實施
PyTorch 的 torch.cuda.amp.autocast 包裝了一個訓練循環,將單個 GPU 上的記憶體大致減半並將吞吐量加倍
在 TPU 上訓練大型語言模型,例如 bfloat16 中的 GPT 式轉換器,以避免損失縮放調整
透過將 ResNet 影像訓練從 FP32 切換到 FP16,在消費級 RTX GPU 上安裝更大的批次大小
NVIDIA H100 GPU 上的 FP8 混合精度可降低預訓練前緣規模模型的成本
混合精準度訓練的未來
精度不斷下降。 NVIDIA Hopper 和 Blackwell GPU 支援的 FP8 訓練正在成為前沿模型的標準,並且對 FP4 和微縮放格式 (MXFP) 的研究也在進一步推進。期望框架能夠自動選擇每層精度,硬體能夠原生處理更窄的格式,量化感知訓練能夠模糊低精度訓練和推理之間的界限,從而降低訓練萬億參數模型的成本。
付諸實踐
為什麼這很重要
- 成本與預算. 多年來,架構決策決定著效能和營運成本。
- 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
- 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。
留意
- 優化一項基準測試可以隱藏更廣泛的系統弱點。
- 基礎設施和維護成本常常被低估。
- 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。
後續步驟
- 在實施之前定義延遲、品質和成本目標。
- 在實際負載和資料條件下進行基準測試。
- 儀器監控錯誤、漂移和使用者影響。
- 在擴展之前準備回滾和事件回應路徑。
不斷探索
常見問題
What is Mixed Precision Training?
Mixed precision uses lower-precision arithmetic for selected operations while retaining precision where needed. It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.
In an FP16 training recipe that keeps an FP32 master weight, why retain it?
Some FP16 recipes keep an FP32 master weight so small optimizer updates are not lost to low-precision rounding; not every mixed-precision recipe uses a separate copy.
FP16 訓練中損失縮放解決了什麼問題?
FP16 的動態範圍有限,因此小梯度可以捨入為零;在反向傳播之前乘以損失可以使它們具有可表示性。
bfloat16相對於FP16有什麼優勢?
Bfloat16 與 FP32 一樣保留了 8 個指數位,因此其動態範圍較大,梯度下溢很少見。
For many supported Tensor Core matrix operations, how are partial sums commonly accumulated?
Many common Tensor Core operations accumulate in FP32 to reduce rounding error; this does not make the computation exact.
What is a common benefit of using 16-bit values for supported training operations?
Lower precision can reduce memory and speed supported operations, but total results depend on the model, hardware, and kernels used.