Technical GUIDE
Mixed Precision Training
Mixed precision uses lower-precision arithmetic for selected operations while retaining precision where needed.
Listen to this guideAI-narrated · ~2 min
On this page2 min read
Overview
It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.
Deep Dive
Traditional training stores weights and runs math in 32-bit floating point (FP32). Mixed precision uses lower-precision FP16 or bfloat16 for eligible operations while retaining FP32 where needed. Some FP16 recipes maintain a separate FP32 master copy to accumulate small weight updates; autocast workflows may instead keep model parameters in FP32 while casting selected operation inputs. Using 16-bit tensors can reduce memory used by eligible operations. Supported accelerators may execute some low-precision operations faster, but end-to-end throughput depends on hardware, workload and which operations autocast selects. FP16 has a narrower range, so tiny gradients can underflow. Loss scaling multiplies the loss before backpropagation and reverses that scale before the optimizer update; tools such as torch.amp.GradScaler can manage this process. Bfloat16 has a wider exponent range than FP16 and often does not need the same loss-scaling strategy, though model and hardware behavior still need testing.
Real-World Implementation
PyTorch's torch.cuda.amp.autocast wrapping a training loop to roughly halve memory and double throughput on a single GPU
Training large language models like GPT-style transformers in bfloat16 on TPUs to avoid loss-scaling tuning
Fitting a larger batch size on a consumer RTX GPU by switching ResNet image training from FP32 to FP16
FP8 mixed precision on NVIDIA H100 GPUs to cut the cost of pretraining frontier-scale models
The Future of Mixed Precision Training
Precision keeps dropping. FP8 training, supported on NVIDIA Hopper and Blackwell GPUs, is becoming standard for frontier models, and research into FP4 and microscaling formats (MXFP) pushes further. Expect frameworks to auto-select per-layer precision, hardware to natively handle ever-narrower formats, and quantization-aware training to blur the line between low-precision training and inference, shrinking the cost of training trillion-parameter models.
Putting it into practice
Why it matters
- Cost and budget. Architecture decisions drive performance and operating cost for years.
- Clearer decisions. Technical education helps teams choose the right stack, not just the newest one.
- Quality control. Better engineering choices reduce reliability incidents in production.
Watch out for
- Optimizing one benchmark can hide broader system weaknesses.
- Infrastructure and maintenance costs are often underestimated.
- Security and observability gaps can grow as systems become more complex.
Next steps
- Define latency, quality, and cost targets before implementation.
- Benchmark under realistic load and data conditions.
- Instrument monitoring for errors, drift, and user impact.
- Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixed Precision Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is Mixed Precision Training?
Mixed precision uses lower-precision arithmetic for selected operations while retaining precision where needed. It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.
In an FP16 training recipe that keeps an FP32 master weight, why retain it?
Some FP16 recipes keep an FP32 master weight so small optimizer updates are not lost to low-precision rounding; not every mixed-precision recipe uses a separate copy.
What problem does loss scaling solve in FP16 training?
FP16 has a limited dynamic range, so small gradients can round to zero; multiplying the loss before backprop keeps them representable.
What advantage does bfloat16 have over FP16?
Bfloat16 keeps 8 exponent bits like FP32, so its dynamic range is large and gradient underflow is rare.
For many supported Tensor Core matrix operations, how are partial sums commonly accumulated?
Many common Tensor Core operations accumulate in FP32 to reduce rounding error; this does not make the computation exact.
What is a common benefit of using 16-bit values for supported training operations?
Lower precision can reduce memory and speed supported operations, but total results depend on the model, hardware, and kernels used.
Keep learning
Related guides
More guides picked for this topic