Nhungamiro yehunyanzvi

Kudzidziswa kweMixed Precision

Mixed precision uses lower-precision arithmetic for selected operations while retaining precision where needed.

Teerera kugwaro iriAI-yakarondedzerwa · ~2 min

Pa peji ino2 min verenga
  1. Pfupiso
  2. Kudzika Kwakadzika
  3. Kuitwa muNyika yeChokwadi
  4. Ramangwana reMixed Precision Training
  5. Kuzvishandisa
  6. Ramba Uchiongorora
  7. Mibvunzo inowanzo bvunzwa

Pfupiso

It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.

Kudzika Kwakadzika

Traditional training stores weights and runs math in 32-bit floating point (FP32). Mixed precision uses lower-precision FP16 or bfloat16 for eligible operations while retaining FP32 where needed. Some FP16 recipes maintain a separate FP32 master copy to accumulate small weight updates; autocast workflows may instead keep model parameters in FP32 while casting selected operation inputs. Using 16-bit tensors can reduce memory used by eligible operations. Supported accelerators may execute some low-precision operations faster, but end-to-end throughput depends on hardware, workload and which operations autocast selects. FP16 has a narrower range, so tiny gradients can underflow. Loss scaling multiplies the loss before backpropagation and reverses that scale before the optimizer update; tools such as torch.amp.GradScaler can manage this process. Bfloat16 has a wider exponent range than FP16 and often does not need the same loss-scaling strategy, though model and hardware behavior still need testing.

Kuitwa muNyika yeChokwadi

PyTorch's torch.cuda.amp.autocast ichiputira loop yekudzidzira kuita hafu yendangariro uye kupinza kaviri paGPU imwe chete.

Kudzidzisa mhando dzemitauro mikuru seGPT-maitiro ekushandura mu bfloat16 paTPU kudzivirira kurasikirwa-kuyera tuning.

Kukodzera saizi yakakura yebatch pamutengi RTX GPU nekuchinja ResNet mufananidzo kudzidziswa kubva kuFP32 kuenda kuFP16.

FP8 yakasanganiswa chaiyo paNVIDIA H100 GPUs yekucheka mutengo wepretraining frontier-scale modhi.

Ramangwana reMixed Precision Training

Precision inoramba ichidonha. FP8 kudzidziswa, inotsigirwa paNVIDIA Hopper uye Blackwell GPUs, yave kuita chiyero chemhando dzemuganhu, uye tsvakiridzo muFP4 uye microscaling mafomati (MXFP) inosundira mberi. Tarisira masisitimu ekusarudza otomatiki pa-layer nemazvo, Hardware kubata anogara-akamanikana mafomati, uye quantization-kuziva kudzidziswa kudzima mutsara pakati peiyo yakaderera-chaiyo kudzidziswa uye kufungidzira, ichideredza mutengo wekudzidzisa matrillion-parameter modhi.

Kuzvishandisa

Nei zvichikosha

  • Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
  • Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
  • Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.

Chenjera

  • Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
  • Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
  • Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.

Matanho anotevera

  1. Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
  2. Benchmark pasi pechokwadi mutoro uye data mamiriro.
  3. Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
  4. Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.

Ramba Uchiongorora

Tsamba yemahara

Tora iyo yemazuva ese AI pfupiso

Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.

Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.

Zviedze iwe pachako

Take the Mixed Precision Training quiz

Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.

Tanga mibvunzo

Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo

Mibvunzo inowanzo bvunzwa

What is Mixed Precision Training?

Mixed precision uses lower-precision arithmetic for selected operations while retaining precision where needed. It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.

In an FP16 training recipe that keeps an FP32 master weight, why retain it?

Some FP16 recipes keep an FP32 master weight so small optimizer updates are not lost to low-precision rounding; not every mixed-precision recipe uses a separate copy.

Idambudziko ripi rinogadziriswa kurasikirwa kuyera muFP16 kudzidziswa?

FP16 ine shoma ine simba renji, saka madiki gradients anogona kutenderera kusvika zero; kuwedzera kurasikirwa pamberi pebackprop inovachengeta vachimiririrwa.

Ndeupi mukana une bfloat16 pamusoro peFP16?

Bfloat16 inochengeta 8 exponent bits seFP32, saka dhizaini yayo yakakura uye gradient underflow haiwanzo.

For many supported Tensor Core matrix operations, how are partial sums commonly accumulated?

Many common Tensor Core operations accumulate in FP32 to reduce rounding error; this does not make the computation exact.

What is a common benefit of using 16-bit values for supported training operations?

Lower precision can reduce memory and speed supported operations, but total results depend on the model, hardware, and kernels used.