Hadin Kai Tsaye horo
Haɗin kai daidaitaccen horo yana haɓaka horon hanyar sadarwar jijiyoyi kuma yana yanke amfani da ƙwaƙwalwar ajiya ta hanyar yin yawancin lissafi a cikin iyo 16-bit maimakon 32-bit.
Dubawa
It lets the same GPU train bigger models faster with almost no loss in accuracy.
Zurfafa nutsewa
Horon al'ada yana adana ma'auni kuma yana gudanar da lissafi a cikin ma'aunin iyo 32-bit (FP32). Haɗe-haɗe daidai yana amfani da ƙananan madaidaicin tsarin 16-bit (FP16 ko bfloat16) don haɓakar matrix mai nauyi, yayin da ke riƙe babban 'kwafin' ma'aunin ma'aunin 32-bit don sabuntawa masu ƙarfi. Saboda lambobi 16-bit suna da rabin girman, sun fi dacewa da ƙwaƙwalwar GPU kuma Tensor Cores suna sarrafa su da sauri 2-8x. Kama shine kunkuntar kewayon FP16: ƙananan gradients na iya shiga cikin sifili. Daidaitaccen daidaitawa shine haɓakar hasara, wanda ke ninka asarar da babban abu kafin yaduwa don haka ƙananan gradients su kasance masu wakilci, sannan su raba shi baya kafin sabuntawar nauyi. NVDIA's Apex da ginanniyar AMP (Aiki da Haɗin Kai ta atomatik) a cikin PyTorch da TensorFlow suna sarrafa wannan.
Fahimtar Fasaha
FP16 yana da juzu'i 5 kawai, yana ba da ƙaramin ƙarfi mai ƙarfi wanda ke haifar da raguwar gradient. Bfloat16 yana adana raƙuman ƙira 8 (daidai da kewayon FP32) amma kaɗan mantissa ragowa, don haka da wuya yana buƙatar hasara - babban dalili Google TPUs da GPUs na zamani sun yarda da shi. Tensor Cores suna haɓaka aikin ta hanyar ninka operands 16-bit amma tara juzu'i a cikin FP32, kiyaye daidaitattun inda kurakuran taƙaitawa zasu iya haɗuwa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Hadin Kai Tsaye
Daidaito yana ci gaba da faduwa. Horon FP8, wanda aka goyan baya akan NVIDIA Hopper da Blackwell GPUs, yana zama daidaitattun samfuran kan iyaka, kuma bincike cikin FP4 da tsarin microscaling (MXFP) yana ci gaba. Yi tsammanin tsarin da za a zaɓa ta atomatik daidaitattun kowane Layer, kayan aikin da za su iya sarrafa tsarin da ke da kunkuntar koyaushe, da horar da ƙididdigewa don ɓata layi tsakanin ƙaramin madaidaicin horo da ƙididdigewa, rage tsadar ƙirar ƙirar tiriliyan.
Aiwatar da Gaskiyar Duniya
PyTorch's Torch.cuda.amp.autocast yana nannade madauki na horo don kusan rabin ƙwaƙwalwar ajiya da fitarwa sau biyu akan GPU guda.
Horar da manyan nau'ikan harshe kamar na'urori masu canzawa irin na GPT a cikin bfloat16 akan TPUs don guje wa daidaita hasara.
Daidaita girman babban tsari akan mabukaci RTX GPU ta hanyar canza horar da hoton ResNet daga FP32 zuwa FP16
FP8 gauraye madaidaicin akan NVIDIA H100 GPUs don rage farashin pretraining-sikelin sikelin iyaka.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixed Precision Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Lakabin Lakabi da Koyarwar Kai
Tambayoyin da ake yawan yi
What is Mixed Precision Training?
Haɗin kai daidaitaccen horo yana haɓaka horon hanyar sadarwar jijiyoyi kuma yana yanke amfani da ƙwaƙwalwar ajiya ta hanyar yin yawancin lissafi a cikin iyo 16-bit maimakon 32-bit. Yana ba da damar GPU iri ɗaya horar da manyan samfura cikin sauri tare da kusan babu asara cikin daidaito.
Me yasa gauraye daidaitattun horo ke kiyaye babban 'kwafin' ma'aunin 32-bit?
Sabunta nauyi sau da yawa kadan ne; tara su cikin 16-bit zai rasa daidaito, don haka cikakken madaidaicin kwafin yana kiyaye sabuntawa daidai.
Wace matsala ɓangarorin asara ke warwarewa a horon FP16?
FP16 yana da iyakataccen kewayo mai ƙarfi, don haka ƙananan gradients na iya zagaye zuwa sifili; ninka hasara kafin backprop ya sa su zama wakilci.
Menene fa'idar bfloat16 akan FP16?
Bfloat16 yana adana ragowa guda 8 kamar FP32, don haka kewayon sa mai ƙarfi yana da girma kuma ƙarancin ƙasa yana da wuya.
Ta yaya Tensor Cores ke kiyaye daidaito yayin amfani da abubuwan shigar 16-bit?
Tensor Cores suna ninka operands 16-bit amma suna taruwa a cikin 32-bit, suna hana kurakuran taƙaitawa daga haɗawa.
Menene fa'idar farko ta amfani da 16-bit maimakon ƙimar 32-bit yayin horo?
Lambobin rabin girman sun dace da ƙarin bayanai a cikin ƙwaƙwalwar GPU kuma su bar ƙwararrun kayan aiki su aiwatar da matrix da sauri sau da yawa.