Incamake
ZeRO (Zero Redundancy Optimizer) ikuraho isesagura ryibikoresho byo kwigana kwigana amakuru kubangikanye no kugabanya leta, gradients, hamwe nuburemere muri GPUs. Iragufasha guhugura moderi nini hamwe nubworoherane bwamakuru ugereranije ariko agace ka memoire ya GPU.
ZeRO na Sharded Optimizers ni tekinoroji yo kubaka tekinike igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo.
Kwibira cyane
Mubisanzwe amakuru abangikanye, buri GPU ibika kopi yuzuye ya optimizer leta, gradients, hamwe nibipimo, birasesagura cyane, cyane cyane kuri Adam, aho leta optimiz ishobora kuba inshuro nyinshi ubunini bwikitegererezo ubwacyo. ZeRO, yatangijwe na Microsoft muri DeepSpeed, ikuraho ubu busumbane igabanya izo tensor muri GPUs kuburyo buri gikoresho gifite igice gusa. ZeRO ije mubyiciro bitatu byiterambere: Icyiciro cya 1 shards optimizer leta, Icyiciro cya 2 kongeramo gradient sharding, naho Stage 3 igabanya ibipimo ubwabyo. Nkuko bikenewe, GPUs ikusanya ibice byabuze ikoresheje itumanaho, kubara, hanyuma ikarekura. Igisubizo kiri hasi cyane yibuka kuri GPU, ituma miliyari- kugeza kuri tiriyari-yamahugurwa, mugihe ukomeje uburyo bworoshye bwo gutangiza amakuru kuburinganire.
Ubushishozi
ZeRO icuruza itumanaho ryinyongera kubitsa kwibuka. Mu cyiciro cya 3, mbere yurwego rwimbere rwimbere, byose-byegeranya bikusanya ibipimo byuzuye kuri buri GPU; nyuma, ibice bitari ibya nyirarureshwa byajugunywe kugirango bigarure kwibuka. Gradients iragabanuka-itatanye kuburyo buri GPU igumana gusa igipimo cya gradient gihuye nibipimo ifite. FSDP ya PyTorch (Yuzuye Data Sharing Data Parallel) ishyira mubikorwa igitekerezo kimwe kavukire, kuzinga modules kuri shard na reshard ku isazi.
Kumenya ZeRO hamwe na Optimizers
ZeRO (Zero Redundancy Optimizer) ikuraho isesagura ryibikoresho byo kwigana kwigana amakuru kubangikanye no kugabanya leta, gradients, hamwe nuburemere muri GPUs. Iragufasha guhugura moderi nini hamwe nubworoherane bwamakuru ugereranije ariko agace ka memoire ya GPU. ZeRO na Sharded Optimizers ni tekinoroji yo kubaka tekinike igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo. Kugirango wubake byimbitse, fata ZeRO na Sharded Optimizers nkicyitegererezo gikora, ntabwo ari ikintu kimwe: gusobanura ibyagezweho, gusobanura ibitekerezo, no gutandukanya ibyo sisitemu ishobora gukora byizewe nibisaba guca imanza zinzobere.
Mubimenyerezo, amakipe akomeye akoresha ZeRO na Sharded Optimizers atezimbere ubwubatsi, amakuru, hamwe nibikorwa remezo birwanya kwizerwa nigiciro. Bandika ibipimo ngenderwaho byerekana intsinzi, bagerageza kurwanya amakuru afatika hamwe nakazi keza, kandi bagasubiramo bashingiye kubikorwa byagaragaye ko batsinzwe aho gutsinda inshuro imwe. Aha niho imyumvire yubumenyi ihinduka mubushobozi burambye kubicuruzwa, politiki, nibikorwa.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mugihe kimwe, Guhindura igipimo kimwe gishobora guhisha intege nke za sisitemu. Uburyo bukomeye cyane ni uguhuza umuvuduko wikigereranyo hamwe na disipuline yimiyoborere: kuyobora abaderevu, gufata ibimenyetso, gutangaza ibyemezo byicyemezo, no gukomeza kuvugurura uburyo bwo kwirinda nkimyitwarire yicyitegererezo, ibyo abakoresha bategereje, nibisabwa n'amategeko bigenda bihinduka.
Ingaruka z'Ingamba
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Gushyira mu bikorwa Isi
Koresha DeepSpeed ZeRO Icyiciro cya 2 kugirango uhuze neza na miriyari-miriyari-yerekana imiterere yururimi rwaba rwuzuye ububiko bwa GPU.
Amahugurwa hamwe na PyTorch FSDP, igabanya ibipimo, gradients, hamwe na optimizer leta muri GPU kandi ikabateranya kuri buri cyiciro kubisabwa.
Gukoresha ZeRO-Offload kugirango usunike optimizer leta mububiko bwa CPU, ureke GPU imwe ihugure icyitegererezo inshuro nyinshi kurenza VRAM yayo.
Gupima moderi ya tiriyari-imwe hamwe na ZeRO-Infinity mugukwirakwiza ibice biva mububiko bwa NVMe mugihe ububiko bwa GPU na CPU burangiye.
Uburyo bwo Gushyira mu bikorwa
ZeRO na Sharded Optimizers mubikorwa
Koresha DeepSpeed ZeRO Icyiciro cya 2 kugirango uhuze neza na miriyari-miriyari-yerekana imiterere yururimi rwaba rwuzuye ububiko bwa GPU.
Gukoresha DeepSpeed ZeRO Icyiciro cya 2 kugirango uhuze neza na miriyari-miriyari-yerekana imiterere yururimi rwaba rwuzuza ubundi urwibutso rwa GPU Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
ZeRO na Sharded Optimizers mubikorwa
Amahugurwa hamwe na PyTorch FSDP, igabanya ibipimo, gradients, hamwe na optimizer leta muri GPU kandi ikabateranya kuri buri cyiciro kubisabwa.
Amahugurwa hamwe na PyTorch FSDP, igabanya ibipimo, gradients, hamwe na optimizer leta muri GPU kandi ikabateranya kuri buri cyiciro kubisabwa Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamuka kwabantu kubibazo bikurikirana, kandi agakurikirana inyungu zibyara umusaruro nibiciro byamakosa mugihe runaka.
ZeRO na Sharded Optimizers mubikorwa
Gukoresha ZeRO-Offload kugirango usunike optimizer leta mububiko bwa CPU, ureke GPU imwe ihugure icyitegererezo inshuro nyinshi kurenza VRAM yayo.
Gukoresha ZeRO-Offload kugirango usunike optimizer leta yibuka CPU, kureka GPU imwe itoza icyitegererezo inshuro nyinshi kurenza Amakipe yayo ya VRAM mubisanzwe ibona ibisubizo byiza mugihe isobanura ibipimo byimbere imbere, igakomeza inzira yo kuzamura abantu kubibazo byimbere, kandi ikurikirana inyungu zibyara umusaruro nibiciro byamakosa mugihe.
ZeRO na Sharded Optimizers mubikorwa
Gupima moderi ya tiriyari-imwe hamwe na ZeRO-Infinity mugukwirakwiza ibice biva mububiko bwa NVMe mugihe ububiko bwa GPU na CPU burangiye.
Gupima moderi ya miriyari hamwe na ZeRO-Infinity mugutondekanya ibipimo biva mububiko bwa NVMe mugihe ububiko bwa GPU na CPU burangiye Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro nibiciro byigihe.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.