Dubawa
Haɗin kai daidaitaccen horo yana haɓaka horon hanyar sadarwar jijiyoyi kuma yana yanke amfani da ƙwaƙwalwar ajiya ta hanyar yin yawancin lissafi a cikin iyo 16-bit maimakon 32-bit. Yana ba da damar GPU iri ɗaya horar da manyan samfura cikin sauri tare da kusan babu asara cikin daidaito.
Haɗaɗɗen Daidaitaccen Horarwa ƙaƙƙarfan tubalin ginin fasaha ne wanda ke shafar ingancin ƙira, farashin kayayyakin more rayuwa, jinkiri, da aminci a sikeli.
Zurfafa nutsewa
Horon al'ada yana adana ma'auni kuma yana gudanar da lissafi a cikin ma'aunin iyo 32-bit (FP32). Haɗe-haɗe daidai yana amfani da ƙananan madaidaicin tsarin 16-bit (FP16 ko bfloat16) don haɓakar matrix mai nauyi, yayin da ke riƙe babban 'kwafin' ma'aunin ma'aunin 32-bit don sabuntawa masu ƙarfi. Saboda lambobi 16-bit suna da rabin girman, sun fi dacewa da ƙwaƙwalwar GPU kuma Tensor Cores suna sarrafa su da sauri 2-8x. Kama shine kunkuntar kewayon FP16: ƙananan gradients na iya shiga cikin sifili. Daidaitaccen daidaitawa shine haɓakar hasara, wanda ke ninka asarar da babban abu kafin yaduwa don haka ƙananan gradients su kasance masu wakilci, sannan su raba shi baya kafin sabuntawar nauyi. NVDIA's Apex da ginanniyar AMP (Aiki da Haɗin Kai ta atomatik) a cikin PyTorch da TensorFlow suna sarrafa wannan.
Fahimtar Fasaha
FP16 yana da juzu'i 5 kawai, yana ba da ƙaramin ƙarfi mai ƙarfi wanda ke haifar da raguwar gradient. Bfloat16 yana adana raƙuman ƙira 8 (daidai da kewayon FP32) amma kaɗan mantissa ragowa, don haka da wuya yana buƙatar hasara - babban dalili Google TPUs da GPUs na zamani sun yarda da shi. Tensor Cores suna haɓaka aikin ta hanyar ninka operands 16-bit amma tara juzu'i a cikin FP32, kiyaye daidaitattun inda kurakuran taƙaitawa zasu iya haɗuwa.
Kwarewar Koyarwar Haɗaɗɗen Daidaitawa
Haɗin kai daidaitaccen horo yana haɓaka horon hanyar sadarwar jijiyoyi kuma yana yanke amfani da ƙwaƙwalwar ajiya ta hanyar yin yawancin lissafi a cikin iyo 16-bit maimakon 32-bit. Yana ba da damar GPU iri ɗaya horar da manyan samfura cikin sauri tare da kusan babu asara cikin daidaito. Haɗaɗɗen Daidaitaccen Horarwa ƙaƙƙarfan tubalin ginin fasaha ne wanda ke shafar ingancin ƙira, farashin kayayyakin more rayuwa, jinkiri, da aminci a sikeli. Don haɓaka fahimta mai zurfi, bi da horon Daidaitaccen Haɗin kai azaman ƙirar aiki, ba fasali ɗaya ba: ayyana sakamakon da ake so, fayyace zato, da raba abin da tsarin zai iya yi da dogaro daga abin da har yanzu ke buƙatar yanke hukunci na ƙwararru.
A aikace, ƙungiyoyi masu ƙarfi da ke amfani da Haɗaɗɗen Koyarwar Daidaitawa suna haɓaka gine-gine, bayanai, da zaɓin abubuwan more rayuwa tare da dogaro da farashi. Suna rubuta ƙayyadaddun ƙa'idodin nasara, gwaji akan bayanan gaskiya da gudanawar aiki, da jujjuyawar bisa ga tsarin gazawar da aka lura maimakon cin nasara na lokaci ɗaya. Wannan shine inda fahimtar ka'idar ta juya zuwa iyawa mai dorewa a cikin samfura, manufofi, da ayyuka.
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru. A lokaci guda, Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin. Hanyar da ta fi dacewa ita ce haɗa saurin gwaji tare da horon gudanarwa: gudanar da matukin jirgi, kama shaida, buga rajistan ayyukan yanke shawara, da ci gaba da sabunta abubuwan tsaro kamar yadda halayen ƙira, tsammanin mai amfani, da buƙatun tsari ke tasowa.
Dabarun Tasiri
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru. A cikin ƙawance masu inganci, ana fassara wannan zuwa ƙa'idodin aiki waɗanda za a iya aunawa, iyakokin ikon mallaka, da kuma bita-da-kullin bita don ƙungiyoyi su iya haɓaka kwarin gwiwa a maimakon ɓata shakku.
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba. A cikin ƙawance masu inganci, ana fassara wannan zuwa ƙa'idodin aiki waɗanda za a iya aunawa, iyakokin ikon mallaka, da kuma bita-da-kullin bita don ƙungiyoyi su iya haɓaka kwarin gwiwa a maimakon ɓata shakku.
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa. A cikin ƙawance masu inganci, ana fassara wannan zuwa ƙa'idodin aiki waɗanda za a iya aunawa, iyakokin ikon mallaka, da kuma bita-da-kullin bita don ƙungiyoyi su iya haɓaka kwarin gwiwa a maimakon ɓata shakku.
Aiwatar da Gaskiyar Duniya
PyTorch's Torch.cuda.amp.autocast yana nannade madauki na horo don kusan rabin ƙwaƙwalwar ajiya da fitarwa sau biyu akan GPU guda.
Horar da manyan nau'ikan harshe kamar na'urori masu canzawa irin na GPT a cikin bfloat16 akan TPUs don guje wa daidaita hasara.
Daidaita girman babban tsari akan mabukaci RTX GPU ta hanyar canza horar da hoton ResNet daga FP32 zuwa FP16
FP8 gauraye madaidaicin akan NVIDIA H100 GPUs don rage farashin pretraining-sikelin sikelin iyaka.
Hanyoyin Aiwatarwa
Hadin Kai Tsaye a aikace
PyTorch's Torch.cuda.amp.autocast yana nannade madauki na horo don kusan rabin ƙwaƙwalwar ajiya da fitarwa sau biyu akan GPU guda.
PyTorch's torch.cuda.amp.autocast nannade madauki na horo don kusan rabin ƙwaƙwalwar ajiya da fitarwa sau biyu akan Ƙungiyoyin GPU guda ɗaya yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓakar ɗan adam don ƙararraki, da bin diddigin abubuwan samarwa da ƙimar kuskure akan lokaci.
Hadin Kai Tsaye a aikace
Horar da manyan nau'ikan harshe kamar na'urori masu canzawa irin na GPT a cikin bfloat16 akan TPUs don guje wa daidaitawa-asara.
Horar da manyan nau'ikan harshe kamar masu canji irin na GPT a cikin bfloat16 akan TPUs don guje wa ƙwalƙwalwar daidaitawa Ƙungiyoyi yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓakar ɗan adam don ƙararraki, da bin diddigin nasarorin samarwa da ƙimar kuskure akan lokaci.
Hadin Kai Tsaye a aikace
Daidaita girman tsari mai girma akan mabukaci RTX GPU ta hanyar sauya horon hoto na ResNet daga FP32 zuwa FP16.
Daidaita girman babban tsari akan mabukaci RTX GPU ta hanyar canza horon hoto na ResNet daga FP32 zuwa Ƙungiyoyin FP16 yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓakar ɗan adam don ƙararraki, da bin diddigin nasarorin samarwa da ƙimar kuskure akan lokaci.
Hadin Kai Tsaye a aikace
FP8 gauraye daidaito akan NVIDIA H100 GPUs don rage farashin pretraining na sikelin sikelin kan iyaka.
FP8 gauraye daidaito akan NVIDIA H100 GPUs don rage farashin pretraining sikelin sikelin ƙididdiga Ƙungiyoyi yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓakar ɗan adam don shari'o'in gefe, da bin duk nasarorin samarwa da farashi na kuskure akan lokaci.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.