ZeRO da Sharded Optimizers
ZeRO (Zero Redundancy Optimizer) yana kawar da ɓarnawar ƙwaƙwalwar ajiyar kwafi na daidaiton bayanai ta hanyar raba yanayin ingantawa, gradients, da ma'auni a cikin GPUs.
Dubawa
Yana ba ku damar horar da ɗimbin ƙira tare da sauƙin daidaita bayanai amma ɗan juzu'in ƙwaƙwalwar kowane-GPU.
Zurfafa nutsewa
A cikin daidaiton bayanai na yau da kullun, kowane GPU yana adana cikakken kwafin yanayin ingantawa, gradients, da sigogi, wanda yake da ɓarna sosai, musamman ga Adamu, inda yanayin ingantawa na iya ninka girman samfurin da kanta. ZeRO, wanda Microsoft ya gabatar a cikin DeepSpeed, yana cire wannan jan aiki ta hanyar rarraba waɗannan tenors a cikin GPUs don haka kowace na'ura ta mallaki yanki guda kawai. ZeRO ya zo cikin matakai masu ci gaba uku: Stage 1 shards optimizer state, Stage 2 yana ƙara gradient sharding, da Stage 3 shards sigogi da kansu. Kamar yadda ake buƙata, GPUs suna tattara ɓangarorin da suka ɓace ta hanyar sadarwa, lissafta, sannan a sake su. Sakamakon yana da matuƙar ƙarancin ƙwaƙwalwar ajiya a kowane GPU, yana ba da damar horar da ma'auni na biliyan- zuwa tiriliyan, yayin da ke kiyaye tsarin tsari mai sauƙi na daidaiton bayanai.
Fahimtar Fasaha
ZeRO yana cinikin ƙarin sadarwa don ajiyar ƙwaƙwalwar ajiya. A mataki na 3, kafin wucewar gaba na Layer, duk wani taro yana tattara cikakkun sigogin Layer akan kowane GPU; daga baya an zubar da ɓangarorin da ba mallakarsu ba don dawo da ƙwaƙwalwar ajiya. Gradients an rage-warwatse don haka kowane GPU yana kiyaye yanki na gradient kawai wanda ya dace da sigogin da ya mallaka. PyTorch's FSDP (Cikakken Sharded Data Parallel) yana aiwatar da ra'ayi iri ɗaya na asali, nannade kayan aiki zuwa shard da sake sakewa akan tashi.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar ZeRO da Sharded Optimizers
Sharding yana zama tsoho don babban horo maimakon zaɓi na ban mamaki. Yi tsammanin haɗin kai mai zurfi tare da ƙaddamarwa (turawa yanka zuwa CPU ko NVMe ta hanyar ZeRO-Infinity), mafi kyawun haɗuwa da tattarawa da rage-watsawa tare da ƙididdigewa don ɓoye farashin su, da haɗuwa tare da tensor da daidaitawar bututun. Kamar yadda samfura ke ci gaba da girma, masu haɓaka ƙwaƙƙwaran ƙwaƙwalwar ajiya sune tsakiya don dacewa da su akan kasafin kayan masarufi na gaske.
Aiwatar da Gaskiyar Duniya
Yin amfani da DeepSpeed Zero Stage 2 don daidaita ƙirar harshe na biliyan-biliyoyin da yawa wanda in ba haka ba zai mamaye ƙwaƙwalwar GPU.
Horowa tare da PyTorch FSDP, wanda ke lalata sigogi, gradients, da yanayin ingantawa a cikin GPUs kuma yana tattara su kowane Layer akan buƙata.
Aiwatar da ZeRO-Offload don tura yanayin ingantawa zuwa ƙwaƙwalwar ajiyar CPU, barin GPU guda ɗaya ya horar da ƙira sau da yawa girma fiye da VRAM ɗin sa.
Ƙirƙirar ƙirar siga- tiriliyan tare da ZeRO-Infinity ta hanyar yawo shards daga ajiyar NVMe lokacin ƙwaƙwalwar GPU da CPU ta ƙare.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the ZeRO and Sharded Optimizers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Lookhead and Lion Optimizers
Tambayoyin da ake yawan yi
Menene ZeRO da Sharded Optimizers?
ZeRO (Zero Redundancy Optimizer) yana kawar da ɓarnawar ƙwaƙwalwar ajiyar kwafi na daidaiton bayanai ta hanyar raba yanayin ingantawa, gradients, da ma'auni a cikin GPUs. Yana ba ku damar horar da manya-manyan ƙira tare da sauƙi na daidaiton bayanai amma ɗan juzu'in ƙwaƙwalwar kowane-GPU.
Menene sakewa ZeRO ke kawar idan aka kwatanta da daidaitattun bayanai?
Daidaitaccen daidaiton bayanai yana adana cikakken kwafin jihar ingantawa, gradients, da ma'auni akan kowane GPU; ZeRO shards waɗannan don haka kowane GPU yana riƙe yanki guda kawai.
Me yasa jihar ingantawa sau da yawa shine mafi girman ƙwaƙwalwar ƙwaƙwalwa tare da Adamu?
Adamu yana kiyaye ƙididdiga masu gudana kamar lokacin farko da na biyu a kowace siga, waɗanda haɗe tare da fp32 ma'aunin nauyi na iya rage girman samfurin.
Menene ZeRO Stage 3 shard wanda Stage 1 da 2 ba sa?
Stage 1 shards optimizer state, Stage 2 yana ƙara gradients, kuma Stage 3 ya ci gaba ta hanyar rarraba sigogin ƙira a cikin GPUs kuma.
A cikin ZeRO Stage 3, ta yaya GPU ke samun cikakkun sigogin da yake buƙata don wucewar gaba?
Kafin lissafta wani Layer, duk mai tarawa yana tattara cikakkun sigoginsa akan kowane GPU; da zarar an yi, ɓangarorin da ba na mallakar su sun sami 'yanci don dawo da ƙwaƙwalwar ajiya.
Wanne fasalin PyTorch ne ke aiwatar da sharding irin na ZeRO?
PyTorch's Fully Sharded Data Parallel (FSDP) shards sigogi, gradients, da ingantawa jihar, tattara da kuma sake tsara su a kan tashi, mai kama da ZeRO.