I-ZeRO kanye ne-Shared Optimizers
I-ZeRO (i-Zero Redundancy Optimizer) iqeda ukuphindaphinda kwenkumbulo okumoshayo kokufana kwedatha ngokwabelana ngesimo se-optimizer, ama-gradient, nezisindo kuwo wonke ama-GPU.
Uhlolojikelele
It lets you train enormous models with the simplicity of data parallelism but a fraction of the per-GPU memory.
I-Deep Dive
Ngokufana kwedatha evamile, yonke i-GPU igcina ikhophi egcwele engasasebenzi yesimo se-optimizer, ama-gradients, namapharamitha, okumosha kakhulu, ikakhulukazi ku-Adamu, lapho isimo se-optimizer singaphinda siphindwe kaningi kunosayizi wemodeli ngokwayo. I-ZeRO, eyethulwe ngu-Microsoft ku-DeepSpeed, isusa lokhu kuphinda kusetshenziswe ngokuhlukanisa lawa ma-tensor kuwo wonke ama-GPU ukuze idivayisi ngayinye ibe nocezu kuphela. I-ZeRO iza ngezigaba ezintathu eziqhubekayo: I-Stage 1 shards optimizer state, Isiteji sesi-2 sengeza i-gradient sharding, kanye ne-Stage 3 shards amapharamitha ngokwawo. Njengoba kudingekile, ama-GPU aqoqa izingcezu ezingekho ngokuxhumana, azibale, bese azikhulula. Umphumela uba inkumbulo ephansi kakhulu nge-GPU ngayinye, okuvumela ukuqeqeshwa kwepharamitha eyizigidigidi kuya kwezigidigidi, kuyilapho kugcinwa imodeli yokuhlela elula yokufana kwedatha.
I-Technical Insight
I-ZeRO ihweba ngokuxhumana okwengeziwe ukuze kugcinwe inkumbulo. Esigabeni sesi-3, ngaphambi kokudlulela phambili kwesendlalelo, iqoqo eliqoqayo liqoqa amapharamitha agcwele alolo ngqimba ku-GPU ngayinye; ngemva kwalokho izingcezu okungezona ezakwakho ziyalahlwa ukuze kubuyiselwe inkumbulo. Ama-gradient ahlakazekile ngokwehliswa ngakho-ke i-GPU ngayinye igcina kuphela ucezu lwegradient olufana namapharamitha engelawo. I-FSDP ye-PyTorch (Fully Sharded Data Parallel) isebenzisa umbono ofanayo ngokwemvelo, igoqa amamojula ukuze ishard futhi iphinde ihlukanise impukane.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Le-ZeRO kanye Nezithuthukisi Ezihlanganisiwe
I-Shading isiba yinto ezenzakalelayo yokuqeqeshwa kwezinga elikhulu kunenketho yangaphandle. Lindela ukuhlanganiswa okujulile nokulayisha (ukusunduza izingcezu ku-CPU noma i-NVMe usebenzisa i-ZeRO-Infinity), ukunqwabelana okungcono kwakho konke kanye nokunciphisa ukusabalalisa ngekhompyutha ukuze ufihle izindleko zakho, kanye nezinhlanganisela ne-tensor ne-pipeline parallelism. Njengoba amamodeli eqhubeka ekhula, izilungiseleli ezishiyiwe ezisebenza kahle inkumbulo zibalulekile ekuzifakeni kubhajethi yehardware engokoqobo.
Ukuqaliswa Komhlaba Wangempela
Ukusebenzisa i-DeepSpeed ZeRO Stage 2 ukulungisa kahle imodeli yolimi yezigidigidi ebingachichima inkumbulo ye-GPU.
Ukuqeqeshwa nge-PyTorch FSDP, ehlukanisa amapharamitha, ama-gradients, kanye nesimo se-optimizer kuwo wonke ama-GPU futhi iwaqoqe isendlalelo ngasinye ngokufunwa.
Ukusebenzisa i-ZeRO-Offload ukuze usunduze isimo se-optimizer kumemori ye-CPU, ukuvumela i-GPU eyodwa iqeqeshe imodeli enkulu ngokuphindwe kaningi kune-VRAM yayo.
Ukukala imodeli yepharamitha eyizigidigidi nge-ZeRO-Infinity ngokusakaza ama-shards epharamitha ukusuka kusitoreji se-NVMe lapho i-GPU nememori ye-CPU iphela.
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the ZeRO and Sharded Optimizers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Lobukahead ne-Lion Optimizers
Imibuzo evame ukubuzwa
What is ZeRO and Sharded Optimizers?
I-ZeRO (i-Zero Redundancy Optimizer) iqeda ukuphindaphinda kwenkumbulo okumoshayo kokufana kwedatha ngokwabelana ngesimo se-optimizer, ama-gradient, nezisindo kuwo wonke ama-GPU. Ikuvumela ukuthi uqeqeshe amamodeli amakhulu ngobulula bokufana kwedatha kodwa ingxenye encane yememori ye-GPU ngayinye.
Ikuphi ukuphinda kuqedwe i-ZeRO uma kuqhathaniswa nokufana kwedatha?
Ukufana kwedatha okujwayelekile kugcina ikhophi egcwele yesimo se-optimizer, ama-gradient, nezisindo kuyo yonke i-GPU; I-ZeRO shards lezi ukuze i-GPU ngayinye ibambe ucezu kuphela.
Kungani i-optimizer state ivame ukuba yingulube enkulu yenkumbulo no-Adamu?
U-Adam ugcina izilinganiso ezisebenzayo ezifana nezikhathi zokuqala nezesibili ngepharamitha ngayinye, ezihlanganiswe nezisindo eziyinhloko ze-fp32 zinganciphisa usayizi wemodeli.
Yini i-ZeRO Stage 3 engayenzanga le Isigaba 1 nesesi-2?
Isimo se-Stage 1 shards optimizer, Isiteji 2 sengeza ama-gradients, futhi Isiteji sesi-3 siqhubekela phambili ngokuhlukanisa amapharamitha emodeli kuwo wonke ama-GPU.
Ku-ZeRO Stage 3, i-GPU iwathola kanjani amapharamitha agcwele ayidingayo ukuze kudlule ungqimba oluya phambili?
Ngaphambi kokwenza ikhompuyutha isendlalelo, iqoqo lonke lihlanganisa imingcele yalo egcwele ku-GPU ngayinye; uma sekwenziwe, izingcezu okungezona ezakhe ziyakhululwa ukuze kubuyiselwe inkumbulo.
Isiphi isici se-PyTorch esisebenzisa ngokuzenzakalelayo ukushaja kwesitayela se-ZeRO?
I-PyTorch's Fully Sharded Data Parallel (FSDP) ishadi amapharamitha, ama-gradients, nesimo se-optimizer, iqoqa futhi yabe kabusha ngokundiza, ifanekisela i-ZeRO.