Incamake
Amahugurwa avanze neza yihutisha imyitozo ya neteur kandi agabanya imikoreshereze yibikorwa mukora imibare myinshi muri 16-bit kureremba aho kuba 32-bit. Kureka GPU imwe igahugura moderi nini byihuse nta gihombo kiboneye.
Amahugurwa avanze neza ni inyubako ya tekiniki yubaka igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo.
Kwibira cyane
Imyitozo gakondo ibika uburemere kandi ikoresha imibare muri 32-bit ireremba (FP32). Ivangavanze rivanze rikoresha imiterere-16-bito (FP16 cyangwa bfloat16) kugwiza matrix iremereye, mugihe ubitse 32-bit 'master kopi' yuburemere kugirango ivugururwe rihamye. Kuberako imibare 16-bit ni kimwe cya kabiri cyubunini, ikwiranye cyane na GPU yibuka na Tensor Cores itunganya hafi 2-8x byihuse. Ifatwa ni intera ntoya ya FP16: uduce duto duto dushobora gutemba kugeza kuri zeru. Igipimo gisanzwe ni igipimo cyo gutakaza, kigwiza igihombo kubintu byinshi mbere yo gusubira inyuma kuburyo gradients ntoya iguma ihagarariwe, hanyuma ikayigabanya inyuma mbere yo kuvugurura ibiro. Apex ya NVIDIA kandi yubatswe muri AMP (Automatic Mixed Precision) muri PyTorch na TensorFlow itangiza ibi.
Ubushishozi
FP16 ifite ibice 5 gusa byerekana, itanga urwego ruto rutera intambwe igenda itera. Bfloat16 ibika ibice 8 byerekana (bihuye nurwego rwa FP32) ariko bito bya mantissa bike, kubwibyo ntibikenera gukenera igihombo - impamvu nyamukuru Google TPU na GPU zigezweho zirabishyigikiye. Tensor Cores yihutisha umurimo mukugwiza 16-bit operands ariko ikusanya amafaranga igice muri FP32, ikabika neza aho amakosa yo guteranya yakwiyongera.
Kumenyereza Amahugurwa avanze neza
Amahugurwa avanze neza yihutisha imyitozo ya neteur kandi agabanya imikoreshereze yibikorwa mukora imibare myinshi muri 16-bit kureremba aho kuba 32-bit. Kureka GPU imwe igahugura moderi nini byihuse nta gihombo kiboneye. Amahugurwa avanze neza ni inyubako ya tekiniki yubaka igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo. Kugira ngo wubake byimbitse, fata Amahugurwa avanze neza nkicyitegererezo gikora, ntabwo ari ikintu kimwe: gusobanura ibyagezweho, gusobanura ibitekerezo, no gutandukanya ibyo sisitemu ishobora gukora byizewe nibisaba guca imanza zinzobere.
Mubimenyerezo, amatsinda akomeye akoresheje Amahugurwa avanze neza atezimbere imyubakire, amakuru, hamwe nibikorwa remezo birwanya kwizerwa nigiciro. Bandika ibipimo ngenderwaho byerekana intsinzi, bagerageza kurwanya amakuru afatika hamwe nakazi keza, kandi bagasubiramo bashingiye kubikorwa byagaragaye ko batsinzwe aho gutsinda inshuro imwe. Aha niho imyumvire yubumenyi ihinduka mubushobozi burambye kubicuruzwa, politiki, nibikorwa.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mugihe kimwe, Guhindura igipimo kimwe gishobora guhisha intege nke za sisitemu. Uburyo bukomeye cyane ni uguhuza umuvuduko wikigereranyo hamwe na disipuline yimiyoborere: kuyobora abaderevu, gufata ibimenyetso, gutangaza ibyemezo byicyemezo, no gukomeza kuvugurura uburyo bwo kwirinda nkimyitwarire yicyitegererezo, ibyo abakoresha bategereje, nibisabwa n'amategeko bigenda bihinduka.
Ingaruka z'Ingamba
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Gushyira mu bikorwa Isi
Itara rya PyTorch.cuda.amp.autocast kuzinga imyitozo kugirango ugabanye hafi kwibuka no kwinjiza kabiri kuri GPU imwe
Guhugura imiterere nini yindimi nka GPT yuburyo bwa transformateur muri bfloat16 kuri TPU kugirango wirinde igihombo-gipima
Guhuza ingano nini kumuguzi RTX GPU uhindura imyitozo ya ResNet ishusho kuva FP32 ikajya kuri FP16
FP8 ivanze neza kuri NVIDIA H100 GPUs kugirango igabanye igiciro cyo kwipimisha imipaka-imipaka
Uburyo bwo Gushyira mu bikorwa
Amahugurwa avanze neza mubikorwa
Itara rya PyTorch.cuda.amp.autocast kuzinga imyitozo kugirango ugabanye hafi kwibuka no kwinjiza kabiri kuri GPU imwe.
Itara rya PyTorch.cuda.amp.autocast gupfunyika imyitozo kugirango ugabanye hafi kwibuka no kwinjiza kabiri mumakipe amwe ya GPU mubusanzwe babona ibisubizo byiza mugihe basobanuye ibipimo byujuje ubuziranenge imbere, bagakomeza inzira yo kuzamura abantu kubibazo, kandi bagakurikirana inyungu zibyara umusaruro nibiciro byigihe.
Amahugurwa avanze neza mubikorwa
Guhugura imiterere nini yindimi nka GPT yuburyo bwa transformateur muri bfloat16 kuri TPU kugirango wirinde guhuza igihombo.
Guhugura imiterere nini yindimi nka GPT-yuburyo bwa transfert muri bfloat16 kuri TPUs kugirango wirinde igihombo-gipimo cyo guhuza Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Amahugurwa avanze neza mubikorwa
Guhuza icyiciro kinini kumuguzi RTX GPU uhindura imyitozo ya ResNet ishusho kuva FP32 ikajya kuri FP16.
Guhuza ingano nini kumuguzi RTX GPU uhindura imyitozo ya ResNet ishusho ya FP32 ikajya mumakipe ya FP16 mubusanzwe ibona ibisubizo byiza mugihe isobanura ibipimo ngenderwaho byimbere imbere, igakomeza inzira yo kuzamura abantu kubibazo, kandi ikurikirana inyungu zibyara umusaruro nibiciro byamakosa mugihe runaka.
Amahugurwa avanze neza mubikorwa
FP8 ivanze neza kuri NVIDIA H100 GPUs kugirango igabanye ikiguzi cyo kwipimisha imipaka yimbere.
FP8 ivanze neza kuri NVIDIA H100 GPUs kugirango igabanye ikiguzi cyo kwipimisha imipaka yerekana imipaka Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.