概述
It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.
深入探讨
Traditional training stores weights and runs math in 32-bit floating point (FP32). Mixed precision uses lower-precision FP16 or bfloat16 for eligible operations while retaining FP32 where needed. Some FP16 recipes maintain a separate FP32 master copy to accumulate small weight updates; autocast workflows may instead keep model parameters in FP32 while casting selected operation inputs. Using 16-bit tensors can reduce memory used by eligible operations. Supported accelerators may execute some low-precision operations faster, but end-to-end throughput depends on hardware, workload and which operations autocast selects. FP16 has a narrower range, so tiny gradients can underflow. Loss scaling multiplies the loss before backpropagation and reverses that scale before the optimizer update; tools such as torch.amp.GradScaler can manage this process. Bfloat16 has a wider exponent range than FP16 and often does not need the same loss-scaling strategy, though model and hardware behavior still need testing.
现实世界的实施
PyTorch 的 torch.cuda.amp.autocast 包装了一个训练循环，将单个 GPU 上的内存大致减半并将吞吐量加倍
在 TPU 上训练大型语言模型，例如 bfloat16 中的 GPT 式转换器，以避免损失缩放调整
通过将 ResNet 图像训练从 FP32 切换到 FP16，在消费级 RTX GPU 上安装更大的批量大小
NVIDIA H100 GPU 上的 FP8 混合精度可降低预训练前沿规模模型的成本
混合精度训练的未来
精度不断下降。 NVIDIA Hopper 和 Blackwell GPU 支持的 FP8 训练正在成为前沿模型的标准，并且对 FP4 和微缩放格式 (MXFP) 的研究也在进一步推进。期望框架能够自动选择每层精度，硬件能够原生处理更窄的格式，量化感知训练能够模糊低精度训练和推理之间的界限，从而降低训练万亿参数模型的成本。
付诸实践
为什么这很重要
- 成本与预算. 多年来，架构决策决定着性能和运营成本。
- 更清晰的判决. 技术教育帮助团队选择正确的堆栈，而不仅仅是最新的堆栈。
- 质量控制. 更好的工程选择可以减少生产中的可靠性事故。
留意
- 优化一项基准测试可以隐藏更广泛的系统弱点。
- 基础设施和维护成本常常被低估。
- 随着系统变得更加复杂，安全性和可观察性差距可能会扩大。
后续步骤
- 在实施之前定义延迟、质量和成本目标。
- 在实际负载和数据条件下进行基准测试。
- 仪器监控错误、漂移和用户影响。
- 在扩展之前准备回滚和事件响应路径。
不断探索
常见问题
What is Mixed Precision Training?
Mixed precision uses lower-precision arithmetic for selected operations while retaining precision where needed. It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.
In an FP16 training recipe that keeps an FP32 master weight, why retain it?
Some FP16 recipes keep an FP32 master weight so small optimizer updates are not lost to low-precision rounding; not every mixed-precision recipe uses a separate copy.
FP16 训练中损失缩放解决了什么问题？
FP16 的动态范围有限，因此小梯度可以舍入为零；在反向传播之前乘以损失可以使它们具有可表示性。
bfloat16相对于FP16有什么优势？
Bfloat16 与 FP32 一样保留了 8 个指数位，因此其动态范围较大，梯度下溢很少见。
For many supported Tensor Core matrix operations, how are partial sums commonly accumulated?
Many common Tensor Core operations accumulate in FP32 to reduce rounding error; this does not make the computation exact.
What is a common benefit of using 16-bit values for supported training operations?
Lower precision can reduce memory and speed supported operations, but total results depend on the model, hardware, and kernels used.