Up tókànItọsọna atẹle
Dosing ti konge AI ati Pharmacokinetics
Imọ-ẹrọ
Imọ Itọsọna
Adalu konge Training
Mixed precision uses lower-precision arithmetic for selected operations while retaining precision where needed.
Gbọ si itọsọna yiiAI-ṣe afihan · ~2 min
Lori iwe yi2 min ka
Akopọ
It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.
Jin Dive
Traditional training stores weights and runs math in 32-bit floating point (FP32). Mixed precision uses lower-precision FP16 or bfloat16 for eligible operations while retaining FP32 where needed. Some FP16 recipes maintain a separate FP32 master copy to accumulate small weight updates; autocast workflows may instead keep model parameters in FP32 while casting selected operation inputs. Using 16-bit tensors can reduce memory used by eligible operations. Supported accelerators may execute some low-precision operations faster, but end-to-end throughput depends on hardware, workload and which operations autocast selects. FP16 has a narrower range, so tiny gradients can underflow. Loss scaling multiplies the loss before backpropagation and reverses that scale before the optimizer update; tools such as torch.amp.GradScaler can manage this process. Bfloat16 has a wider exponent range than FP16 and often does not need the same loss-scaling strategy, though model and hardware behavior still need testing.
Real-World imuse
PyTorch's torch.cuda.amp.autocast n murasilẹ lupu ikẹkọ si aijọju iranti idaji ati ilọpo meji lori GPU kan
Ikẹkọ awọn awoṣe ede nla bii awọn oluyipada ara-GPT ni bfloat16 lori awọn TPU lati yago fun yiyi iwọn-pipadanu
Ni ibamu iwọn ipele ti o tobi julọ lori olumulo RTX GPU nipa yiyipada ikẹkọ aworan ResNet lati FP32 si FP16
FP8 dapọ konge lori NVIDIA H100 GPUs lati ge idiyele ti iṣaju iṣaju awọn awoṣe iwọn-aala
Ọjọ iwaju ti Ikẹkọ Itọka Ajọpọ
Itọkasi n tẹsiwaju silẹ. Ikẹkọ FP8, ni atilẹyin lori NVIDIA Hopper ati Blackwell GPUs, n di idiwọn fun awọn awoṣe aala, ati iwadii sinu FP4 ati awọn ọna kika microscaling (MXFP) titari siwaju. Reti awọn ilana lati yan adaṣe-itọkasi-Layer kọọkan, ohun elo ohun elo lati mu ni abinibi mu awọn ọna kika ti o dín nigbagbogbo, ati ikẹkọ ti o mọye lati di laini laini laarin ikẹkọ konge kekere ati itọkasi, idinku idiyele ti awọn awoṣe paramita aimọye-imọye.
Fifi o sinu iwa
Kini idi ti o ṣe pataki
- Iye owo ati isuna. Awọn ipinnu faaji ṣe awakọ iṣẹ ati idiyele iṣẹ fun awọn ọdun.
- Awọn ipinnu diẹ sii. Ẹkọ imọ-ẹrọ ṣe iranlọwọ fun awọn ẹgbẹ lati yan akopọ to tọ, kii ṣe ọkan tuntun nikan.
- Iṣakoso didara. Awọn yiyan imọ-ẹrọ to dara julọ dinku awọn iṣẹlẹ igbẹkẹle ni iṣelọpọ.
Ṣọra fun
- Ṣiṣepe ala-ilẹ kan le tọju awọn ailagbara eto ti o gbooro.
- Awọn ohun elo amayederun ati awọn idiyele itọju nigbagbogbo ni aibikita.
- Aabo ati awọn ela akiyesi le dagba bi awọn eto ṣe di eka sii.
Next awọn igbesẹ
- Ṣetumo lairi, didara, ati awọn ibi-afẹde idiyele ṣaaju imuse.
- Aṣepari labẹ ẹru ojulowo ati awọn ipo data.
- Abojuto ohun elo fun awọn aṣiṣe, fiseete, ati ipa olumulo.
- Mura ipadasẹhin pada ati awọn ipa ọna esi iṣẹlẹ ṣaaju iwọn.
Tesiwaju Ṣiṣawari
Ìwé ìròyìn ọfẹ
Gba ìkànìyàn AI ojoojúmọ́
Ìtàn AI mẹta tó jẹ́rìí ní gbogbo owurọ́ ọjọ́ iṣẹ́, kọ́wé ní Gẹ̀ẹ́sì tó rọrùn. Ọfẹ́ fún gbogbo àkókò, kò sí ìpolówó.
Ìmeèlì kan ní gbogbo ọjọ́ iṣẹ́. Yọkúrò ní ìkànìyàn ní ìkànìyàn kan. A kò ta tàbí pín àdírẹ́sì rẹ.
Ṣe idanwo fun ara rẹ
Take the Mixed Precision Training quiz
Ìdáhùn lẹ́sẹ̀kẹsẹ̀ lórí gbogbo ìdáhùn, àti iwe-ẹ̀rí tó lè pin pẹ̀lú ID tó lè jẹ́rìí lẹ́yìn tí o bá kọ́ ẹ̀kọ́.
Ṣe àtìlẹ́yìn ẹ̀kọ́ AI ọfẹ́. AI Understanding jẹ́ àjọ àìní èrè 501(c)(3) — kò sí ìpolówó, kò sí àkúnya owó, ní gbogbo àkókò. Ṣe ẹbun kan
Awọn ibeere ti a beere nigbagbogbo
What is Mixed Precision Training?
Mixed precision uses lower-precision arithmetic for selected operations while retaining precision where needed. It can reduce memory use and improve throughput on supported hardware, but gains and numerical effects depend on the model, workload, hardware and training recipe.
In an FP16 training recipe that keeps an FP32 master weight, why retain it?
Some FP16 recipes keep an FP32 master weight so small optimizer updates are not lost to low-precision rounding; not every mixed-precision recipe uses a separate copy.
Iṣoro wo ni igbelowọn pipadanu yanju ni ikẹkọ FP16?
FP16 ni iwọn agbara to lopin, nitorinaa awọn gradients kekere le yika si odo; isodipupo isonu ṣaaju ki o to backprop ntọju wọn aṣoju.
Kini anfani ni bfloat16 lori FP16?
Bfloat16 ṣe itọju awọn iwọn 8 olupilẹṣẹ bii FP32, nitorinaa iwọn agbara rẹ tobi ati ṣiṣan aladiẹ jẹ toje.
For many supported Tensor Core matrix operations, how are partial sums commonly accumulated?
Many common Tensor Core operations accumulate in FP32 to reduce rounding error; this does not make the computation exact.
What is a common benefit of using 16-bit values for supported training operations?
Lower precision can reduce memory and speed supported operations, but total results depend on the model, hardware, and kernels used.
Tesiwaju kikọ
Jẹmọ awọn itọsọna
Awọn itọsọna diẹ sii ti a yan fun koko yii