Ngwakọta nkenke Ọzụzụ
Ọzụzụ ziri ezi agwakọta na-eme ka ọzụzụ netwọkụ akwara dị ngwa ma belata ojiji ebe nchekwa site na ịme ọtụtụ mgbakọ na mwepụ na 16-bit floating point kama 32-bit.
Nchịkọta
It lets the same GPU train bigger models faster with almost no loss in accuracy.
Ime miri emi
Ọzụzụ ọdịnala na-echekwa nha ma na-agba mgbakọ na mwepụ na 32-bit floating point (FP32). Ngwakọta agwakọta na-eji usoro 16-bit nkenke dị ala (FP16 ma ọ bụ bfloat16) maka ịba ụba matriks dị arọ, ebe ị na-edobe 'mbibi ukwu' nke 32-bit maka mmelite kwụsiri ike. N'ihi na ọnụọgụ 16-bit bụ ọkara nha, dabara adaba na ebe nchekwa GPU yana Tensor Cores na-ahazi ha ngwa ngwa 2-8x. Ọkụ ahụ bụ FP16 dị warara: obere gradients nwere ike ịbanye na efu. Ọkọlọtọ ndozi bụ ọnwụ scaling, nke na-amụba ọnwụ site nnukwu ihe tupu backpropagation ka obere gradients na-anọchi anya, wee kewaa ya azụ tupu ibu mmelite. NVIDIA's Apex na AMP arụnyere n'ime ya (Automatic Mixed Precision) na PyTorch na TensorFlow na-emezi nke a.
Nghọta nka nka
FP16 nwere naanị 5 exponent ibe n'ibe, na-enye obere ike nso na-akpata gradient eruba. Bfloat16 na-edobe 8 exponent ibe n'ibe (dakọtara na FP32 si nso) ma ole na ole mantissa ibe n'ibe, ya mere ọ na-adịkarịghị mkpa ọnwụ scaling - isi ihe kpatara Google TPUs na ọgbara ọhụrụ GPUs kwadoro ya. Tensor Cores na-eme ka ọrụ ahụ dịkwuo elu site n'ịba ụba operands 16-bit mana na-agbakọta nchikota akụkụ na FP32, na-echekwa nkenke ebe njehie nchịkọta ga-agbakọta ma ọ bụghị ya.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke Ọzụzụ Ngwakọta Ngwakọta
Nkenkenke na-aga n'ihu na-agbada. Ọzụzụ FP8, nke akwadoro na NVIDIA Hopper na Blackwell GPUs, na-aghọ ọkọlọtọ maka ụdị oke, na nyocha n'ime ụdị FP4 na microscaling (MXFP) na-aga n'ihu. Na-atụ anya ka usoro iji họrọ nkenke nkenke nke ọ bụla, ngwaike iji na-ejizi usoro ndị na-adịwanye warara nke ọma, yana ọzụzụ mara nke ọma iji mebie ahịrị n'etiti ọzụzụ dị obere na ntinye, na-ebelata ọnụ ahịa ọzụzụ ụdị trillion-parameter.
Mmejuputa n'ezie n'ụwa
PyTorch's torch.cuda.amp.autocast na-ekechi loop ọzụzụ iji belata ebe nchekwa dị obere na ntinye ugboro abụọ na otu GPU.
Ọzụzụ ụdị asụsụ buru ibu dị ka ihe ntụgharị ụdị GPT na bfloat16 na TPU ka ịzena ntuzigharị ihe na-efu.
Itinye nnukwu batch buru ibu na onye ahịa RTX GPU site na ịgbanwere ọzụzụ onyonyo ResNet site na FP32 gaa na FP16
Ngwakọta FP8 ziri ezi na NVIDIA H100 GPUs iji belata ọnụ ahịa ụzọ ọzụzụ ụdịdị oke ala.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixed Precision Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Ịkpọ aha ụgha na ọzụzụ onwe
Ajụjụ a na-ajụkarị
What is Mixed Precision Training?
Ọzụzụ ziri ezi agwakọta na-eme ka ọzụzụ netwọkụ akwara dị ngwa ma belata ojiji ebe nchekwa site na ịme ọtụtụ mgbakọ na mwepụ na 16-bit floating point kama 32-bit. Ọ na-ahapụ otu GPU ịzụ nnukwu ụdị ngwa ngwa na ọ fọrọ nke nta ka ọ bụrụ enweghị mfu na izi ezi.
Kedu ihe kpatara ọzụzụ izizi agwakọtara na-edobe 32-bit 'mbibi akwụkwọ' nke ịdị arọ?
Mmelite ibu na-adịkarị obere; Ịchịkọta ha na 16-bit ga-efunahụ nkenke, yabụ nnomi nna ukwu zuru oke na-eme ka mmelite dị mma.
Kedu nsogbu mbelata mfu na-edozi na ọzụzụ FP16?
FP16 nwere oke ike dị oke oke, yabụ obere gradients nwere ike ịgbaruo efu; ịba ụba ọnwụ tupu backprop na-eme ka ha bụrụ ndị a na-anọchi anya ya.
Kedu uru bfloat16 nwere karịa FP16?
Bfloat16 na-edobe 8 exponent ibe n'ibe dị ka FP32, ya mere ike ya ike dị ukwuu na gradient underflow bụ obere.
Kedu ka Tensor Cores si echekwa izizi mgbe ị na-eji ntinye 16-bit?
Tensor Cores na-amụba operands 16-bit mana na-agbakọta na 32-bit, na-egbochi njehie nchịkọta site na ịgbakọta.
Kedu uru bụ isi nke iji 16-bit kama ụkpụrụ 32-bit n'oge ọzụzụ?
Ọnụọgụ nke ọkara dabara data karịa na ebe nchekwa GPU wee hapụ usoro mgbakọ na mwepụ ngwaike pụrụ iche ọtụtụ ugboro ọsọ ọsọ.