ZeRO na Sharded Optimizers
ZeRO (Zero Redundancy Optimizer) na-ewepụ myigharị ebe nchekwa na-emebi emebi nke myirịta data site na ịkesa steeti optimizer, gradients, na arọ gafee GPUs.
Nchịkọta
It lets you train enormous models with the simplicity of data parallelism but a fraction of the per-GPU memory.
Ime miri emi
Na ndakọrịta data nkịtị, GPU ọ bụla na-echekwa nnomi zuru oke nke steeti optimizer, gradients na paramita, nke na-emebi emebi, ọkachasị maka Adam, ebe steeti optimizer nwere ike ịdị ọtụtụ ugboro karịa ụdị n'onwe ya. ZeRO, nke Microsoft webatara na DeepSpeed , na-ewepụ ọrụ ọrụ a site na ikewa tenors ndị a n'ofe GPU ka ngwaọrụ ọ bụla nwee naanị iberi. ZeRO na-abịa na usoro atọ na-aga n'ihu: Stege 1 shards optimizer state, Stage 2 na-agbakwụnye gradient sharding, na Stage 3 shards parampat n'onwe ha. Dị ka ọ dị mkpa, GPU na-achịkọta mpekere efu site na nzikọrịta ozi, gbakọọ, wee hapụ ha. Nsonaazụ bụ ebe nchekwa dị ala nke ukwuu n'otu GPU, na-enyere ọzụzụ ijeri na ijeri trillion, na-edobe usoro mmemme dị mfe nke myirịta data.
Nghọta nka nka
ZeRO na-azụta nzikọrịta ozi maka nchekwa nchekwa. N'ọkwa nke 3, tupu ngafe n'ihu oyi akwa, ndị otu niile na-anakọta oke oyi akwa ahụ na GPU nke ọ bụla; emesia a na-atụfu mpekere ndị na-enweghị ka ha nwetaghachi ebe nchekwa. A na-agbasasịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsịsị akwụkwọ) nke ha mere GPU ọ bụla na-edobe nanị iberi gradient dabara na paramita ọ nwere. PyTorch's FSDP (Fully Sharded Data Parallel) na-eme otu echiche ahụ n'ụdị ala, na-ekechi modul ka shard na reshard na ofufe.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke ZeRO na Sharded Optimizers
Sharding na-aghọ ihe ndabere maka ọzụzụ buru ibu karịa nhọrọ pụrụ iche. Na-atụ anya njikọta miri emi na mbugharị (ịkwanye mpekere na CPU ma ọ bụ NVMe site na ZeRO-Infinity), nchikota kacha mma nke mkpokọta ma belata-gbasaa na mgbakọ iji zoo ọnụ ahịa ha, yana njikọta na tensor na pipeline parallelism. Dị ka ụdị na-aga n'ihu na-eto eto, ndị na-ebupụta ihe njiri mara nke ọma bụ ebe nchekwa dị mkpa iji kwado ha na mmefu ego ngwaike dị adị.
Mmejuputa n'ezie n'ụwa
Iji DeepSpeed ZeRO Stage 2 dị mma iji dozie ụdị asụsụ ọtụtụ ijeri ijeri nke ga-ejupụta ebe nchekwa GPU.
Ọzụzụ ya na PyTorch FSDP, nke na-akụda paramita, gradients, na steeti njikarịcha n'ofe GPUs ma na-achịkọta ha kwa oyi akwa mgbe achọrọ.
Itinye ZeRO-Offload ka ịkwanye steeti optimizer na ebe nchekwa CPU, na-ahapụ otu GPU ka ọ zụọ ihe nlereanya ọtụtụ ugboro karịa VRAM ya.
Iji ZeRO-Infinity na-eme ka ihe atụ trillion-parameter na-eme ihe site na ịkwanye shards parameter sitere na nchekwa NVMe mgbe ebe nchekwa GPU na CPU kwụsịrị.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the ZeRO and Sharded Optimizers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Lookhead na Lion Optimizers
Ajụjụ a na-ajụkarị
What is ZeRO and Sharded Optimizers?
ZeRO (Zero Redundancy Optimizer) na-ewepụ myigharị ebe nchekwa na-emebi emebi nke myirịta data site na ịkesa steeti optimizer, gradients, na arọ gafee GPUs. Ọ na-enye gị ohere ịzụ nnukwu ụdị n'ịdị mfe nke data myirịta mana akụkụ dị nta nke ebe nchekwa GPU ọ bụla.
Kedu ihe nkwụghachi ụgwọ ZeRO na-ewepụ ma e jiri ya tụnyere myirịta data doro anya?
Ọkọlọtọ data myirịta na-echekwa nnomi zuru oke nke steeti njikarịcha, gradients, na ihe ọ̀tụ̀tụ̀ na GPU ọ bụla; ZeRO na-agbaji ihe ndị a ka GPU nke ọ bụla na-ejide naanị iberi.
Kedu ihe kpatara steeti optimizer na-abụkarị nnukwu ebe nchekwa na Adam?
Adam na-ejigide atụmatụ na-agba ọsọ dị ka oge mbụ na nke abụọ n'otu paramita, nke jikọtara ya na fp32 nnukwu arọ nwere ike imebi nha nke ihe nlereanya ahụ.
Kedu ihe ZeRO Stage 3 shard nke 1 na 2 na-emeghị?
Steeti 1 shards optimizer steeti, ogbo 2 na-agbakwunye gradients, yana ọkwa nke 3 na-aga n'ihu site n'ịkwalite usoro ihe atụ n'ofe GPUs.
Na ZeRO Stage 3, kedu ka GPU si enweta paramita zuru oke ọ chọrọ maka ngafe n'ihu oyi akwa?
Tupu ịgbakọ oyi akwa, onye na-achịkọta ihe niile na-achịkọta paramita ya zuru oke na GPU ọ bụla; ozugbo emechara, mpekere ndị na-abụghị ndị nwere onwe ha ka ha weghachi ebe nchekwa.
Kedu njiri mara PyTorch na-arụ ọrụ sharding ụdị ZeRO?
PyTorch's Fully Sharded Data Parallel (FSDP) shards parameters, gradients, and optimizer state, na-achịkọta ma na-emegharị ha na ofufe, na-egosipụta ZeRO.