Mbudata steeti Optimizer na CPU na NVMe
Aghụghọ na-echekwa ebe nchekwa nke na-adọba nnukwu nchekwa nke ọzụzụ (steeti ndị na-ebuwanye ibu, gradients, mgbe ụfọdụ ibu) na CPU RAM ma ọ bụ na NVMe SSDs kama ebe nchekwa GPU dị ụkọ.
Nchịkọta
It lets people train far larger models than their GPU's memory would otherwise allow.
Ime miri emi
Mgbe ị na-azụ netwọkụ akwara ozi na ihe na-eme ka Adam dị ka Adam, oke ọ bụla na-ebu ibu ọzọ: ọnụ ọgụgụ na-agba ọsọ abụọ (oge na mgbanwe), gbakwunyere nnomi zuru oke nke ịdị arọ, yana gradient ya. N'ọzụzụ ziri ezi agwakọtara nke a nwere ike ịgbakọta ihe ruru 16 bytes kwa oke, na-akụda bytes abụọ maka ibu ahụ n'onwe ya. Nbudata na-ebupụ akpa ahụ na GPU. CPU offload iyi optimizer na-ekwu n'ime usoro RAM nkịtị n'elu bọs PCIe, ebe NVMe na-ebuli ha na-agbada ruo diski steeti siri ike ngwa ngwa. N'ịbụ nke DeepSpeed's ZeRO-Infinity na ZeRO-Offload na-ewu ewu, usoro a na-azụ ahịa ngwa ngwa maka ikike, na-ahapụ otu GPU ma ọ bụ obere ụyọkọ nke ọma ụdị nwere ọtụtụ ijeri paramita.
Nghọta nka nka
Isi ihe bụ iji mgbako na-agbakọ mmegharị data. Steeti Optimizer na-anọdụ ala na CPU/NVMe; n'oge ngafe azụ, a na-ebufe akụkụ n'elu PCIe tupu achọrọ ha na nzọụkwụ optimizer n'onwe ya na-agbakarị na CPU. ZeRO-Offload na-edobe float32 nna ukwu na oge Adam na CPU, yabụ naanị mgbakọ na mwepụ na-aga n'ihu na azụ na-anọ na GPU. NVMe na-agbakwunye cache nwere ọkwa ka steeti terabyte wee wụsa na diski ebe akụkụ dị ọkụ na-anọ na RAM.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke mbubata steeti Optimizer na CPU na NVMe
Dị ka ụdị na-aga n'ihu na-etowanye ebe nchekwa GPU, mwepu oke na-aghọ ọkọlọtọ kama ọ bụghị ihe pụrụ iche. Na-atụ anya njikọta siri ike na njikọ njikọ ngwa ngwa dị ka ebe nchekwa NVLink-C2C na CXL nke na-eme ka ókèala CPU-GPU, gbakwunyere ndị na-ahazi nhazi nke ọma na-ebu amụma nke ga-ebu ụzọ. Ihe owuwu ebe nchekwa dị n'otu dị ka Grace Hopper na-ebelata ntaramahụhụ PCIe, na usoro na-akwalite ime ka ọtụtụ ihe na-ebufe ihe fọrọ nke nta ka ọ bụrụ nke ọma ka ndị na-enwe ntụrụndụ nwee ike idozi nnukwu ụdị na ngwaike dị ala.
Mmejuputa n'ezie n'ụwa
Nhazi nke ọma 13-ijeri-iri LLM na otu 24 GB nke ndị na-azụ ahịa GPU site na iji DeepSpeed Zero-Offload iji kwanye steeti Adam na CPU RAM.
Obere ụlọ nyocha nyocha na-azụ ụdị ihe atụ ijeri ijeri na GPU ole na ole site na ịwụfu steeti optimizer na draịva NVMe na ZeRO-Infinity.
Ịmakụ ihu na-eme ngwa ngwa nhazi nke na-eme ka CPU wepụ ya ka ndị ọrụ nwee ike ịrụ ọrụ nhazi nke ọma nke ga-atụfu njehie na-enweghị ncheta.
Ndị mbido na-akwụ ụgwọ na-akwụ ụgwọ dị ọnụ ala, GPU igwe ojii dị ala ma na-ebufe na NVMe agbakwunyere kama ịkwụ ụgwọ maka kaadị 80 GB dị elu.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Optimizer State Offloading to CPU and NVMe quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Adam na ndị na-eme mgbanwe
Ajụjụ a na-ajụkarị
What is Optimizer State Offloading to CPU and NVMe?
Aghụghọ na-echekwa ebe nchekwa nke na-adọba nnukwu nchekwa nke ọzụzụ (steeti ndị na-ebuwanye ibu, gradients, mgbe ụfọdụ ibu) na CPU RAM ma ọ bụ na NVMe SSDs kama ebe nchekwa GPU dị ụkọ. Ọ na-ahapụ ndị mmadụ ka ha zụọ ụdịdị ka ukwuu karịa ebe nchekwa GPU ha ga-enye ma ọ bụghị.
Kedu ebumnuche bụ isi nke ibupụ steeti optimizer na CPU ma ọ bụ NVMe?
Nbudata na-ebupụ ihe nrụpụta dị arọ na GPU n'ime CPU RAM ma ọ bụ NVMe, na-ahapụ ebe nchekwa GPU ka e wee nwee ike ịzụ ụdị ndị buru ibu n'otu ngwaike.
Maka Adam optimizer n'ụzọ ziri ezi, kedu data na-erikarị ebe nchekwa kachasị n'otu oke?
Adam na-echekwa oge, mgbanwe, na ịdị arọ nna ukwu zuru oke, na-agbakọta ihe dị ka bytes 16 n'otu oke, karịa oke ọkara-byte nke ziri ezi.
Kedu ihe njiri mara mma na-ebuwanye ibu ibu ibu?
DeepSpeed's ZeRO-Offload na ZeRO-Infinity webatara ma kwalite steeti nkwalite mbupu na CPU na NVMe n'ogo.
Kedu ụgwọ ọrụ bụ isi nke ibufe na CPU ma ọ bụ NVMe?
Mbugharị steeti kwụsịrị-GPU pụtara ibufe data n'elu PCIe ma ọ bụ na NVMe, nke na-eji nwayọ karịa ebe nchekwa GPU, yabụ ntinye na-adaba ọ gwụla ma ejiri mgbako kpuchie ya.
Kedu ka sistemu mbupụ si ezochi ọtụtụ oge mbufe?
Ndị na-ahazi nhazi na-ebute ụzọ dị mkpa n'elu PCIe ka GPU ka na-agbakọ, na-ejikọta nkwurịta okwu na mgbakọ na-ekpuchi anya.