ZeRO na Optimizers
ZeRO (Zero Redundancy Optimizer) ikuraho isesagura ryibikoresho byo kwigana kwigana amakuru kubangikanye no kugabanya leta, gradients, hamwe nuburemere muri GPUs.
Incamake
It lets you train enormous models with the simplicity of data parallelism but a fraction of the per-GPU memory.
Kwibira cyane
Mubisanzwe amakuru abangikanye, buri GPU ibika kopi yuzuye ya optimizer leta, gradients, hamwe nibipimo, birasesagura cyane, cyane cyane kuri Adam, aho leta optimiz ishobora kuba inshuro nyinshi ubunini bwikitegererezo ubwacyo. ZeRO, yatangijwe na Microsoft muri DeepSpeed, ikuraho ubu busumbane igabanya izo tensor muri GPUs kuburyo buri gikoresho gifite igice gusa. ZeRO ije mubyiciro bitatu byiterambere: Icyiciro cya 1 shards optimizer leta, Icyiciro cya 2 kongeramo gradient sharding, naho Stage 3 igabanya ibipimo ubwabyo. Nkuko bikenewe, GPUs ikusanya ibice byabuze ikoresheje itumanaho, kubara, hanyuma ikarekura. Igisubizo kiri hasi cyane yibuka kuri GPU, ituma miliyari- kugeza kuri tiriyari-yamahugurwa, mugihe ukomeje uburyo bworoshye bwo gutangiza amakuru kuburinganire.
Ubushishozi
ZeRO icuruza itumanaho ryinyongera kubitsa kwibuka. Mu cyiciro cya 3, mbere yurwego rwimbere rwimbere, byose-byegeranya bikusanya ibipimo byuzuye kuri buri GPU; nyuma, ibice bitari ibya nyirarureshwa byajugunywe kugirango bigarure kwibuka. Gradients iragabanuka-itatanye kuburyo buri GPU igumana gusa igipimo cya gradient gihuye nibipimo ifite. FSDP ya PyTorch (Yuzuye Data Sharing Data Parallel) ishyira mubikorwa igitekerezo kimwe kavukire, kuzinga modules kuri shard na reshard ku isazi.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza ha ZeRO na Sharing Optimizers
Sharding ihinduka nkibisanzwe mumahugurwa manini aho kuba exotic option. Tegereza kwishyira hamwe kwimbitse (gusunika ibice kuri CPU cyangwa NVMe unyuze kuri ZeRO-Infinity), byiza guhuzagurika byose hamwe no kugabanya-gutatanya hamwe no kubara kugirango uhishe ikiguzi cyabyo, hamwe no guhuza tensor hamwe no guhuza imiyoboro. Nka moderi ikomeza kwiyongera, optimizers yibikoresho ikora neza nibyingenzi kugirango ihuze na bije yibikoresho bifatika.
Gushyira mu bikorwa Isi
Koresha DeepSpeed ZeRO Icyiciro cya 2 kugirango uhuze neza na miriyari-miriyari-yerekana imiterere yururimi rwaba rwuzuye ububiko bwa GPU.
Amahugurwa hamwe na PyTorch FSDP, igabanya ibipimo, gradients, hamwe na optimizer leta muri GPU kandi ikabateranya kuri buri cyiciro kubisabwa.
Gukoresha ZeRO-Offload kugirango usunike optimizer leta mububiko bwa CPU, ureke GPU imwe ihugure icyitegererezo inshuro nyinshi kurenza VRAM yayo.
Gupima moderi ya tiriyari-imwe hamwe na ZeRO-Infinity mugukwirakwiza ibice biva mububiko bwa NVMe mugihe ububiko bwa GPU na CPU burangiye.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the ZeRO and Sharded Optimizers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Reba neza na Ntare
Ibibazo bikunze kubazwa
What is ZeRO and Sharded Optimizers?
ZeRO (Zero Redundancy Optimizer) ikuraho isesagura ryibikoresho byo kwigana kwigana amakuru kubangikanye no kugabanya leta, gradients, hamwe nuburemere muri GPUs. Iragufasha guhugura moderi nini hamwe nubworoherane bwamakuru ugereranije ariko agace ka memoire ya GPU.
Ni ubuhe busumbane ZeRO ikuraho ugereranije namakuru asobanutse?
Ubusanzwe amakuru abangikanye abika kopi yuzuye ya optimizer leta, gradients, hamwe nuburemere kuri buri GPU; ZeRO isatura ibi kuburyo buri GPU ifata igice gusa.
Ni ukubera iki optimizer leta ikunda kwibuka cyane hamwe na Adam?
Adam akomeza kugereranya nkibihe byambere nubwa kabiri kuri parameter, ihujwe nuburemere bwa fp32 bushobora kugabanya urugero rwicyitegererezo.
Niki ZeRO Icyiciro cya 3 shard Icyiciro cya 1 nicya 2 bidakora?
Icyiciro cya 1 shards optimizer leta, Icyiciro cya 2 kongeramo gradients, naho icyiciro cya 3 kijya kure mukugabanya ibipimo byicyitegererezo muri GPUs.
Muri ZeRO Icyiciro cya 3, nigute GPU ibona ibipimo byuzuye ikenera kugirango igere imbere?
Mbere yo kubara igipande, byose-byegeranya biteranya ibipimo byuzuye kuri buri GPU; bimaze gukorwa, ibice bitari ibya nyirubwite birekuwe kugirango bigarure kwibuka.
Nibihe PyTorch biranga kavukire bishyira mubikorwa bya ZeRO?
PyTorch's Sharled Data Data Parallel (FSDP) yerekana ibipimo, gradients, hamwe na optimizer leta, gukusanya no kubisubiza hejuru yisazi, bikerekana ZeRO.