U-dejinta Gobolka Optimizer ee CPU iyo NVMe
Khiyaamo kaydinta xusuusta oo dhigata kaydinta culus ee tababarka (gobolada hagaajinta, gradients, mararka qaarkood miisaan) CPU RAM ama NVMe SSDs halkii ay ka ahaan lahaayeen xusuusta GPU-da yar.
Dulmar
It lets people train far larger models than their GPU's memory would otherwise allow.
quusid qoto dheer
Markaad tababarto shabakad neural ah oo leh hagaajin sida Adam oo kale, halbeeg kastaa wuxuu sido shandad dheeri ah: laba tirakoob oo socda (xoog iyo kala duwanaansho), oo lagu daray koobi sax ah oo miisaan ah, oo lagu daray gradientkiisa. Tababarka saxda ah ee isku dhafan tani waxay wadarta guud ahaan kartaa 16 bytes halkiibeegba, iyada oo ka dhigaysa 2 bytes miisaanka laftiisa. Soo dejintu waxay ka rarisaa alaabtaas GPU-da. CPU offload durdurrada kor u qaadaya gobolada nidaamka caadiga ah RAM ee baska PCIe, halka NVMe offload iyaga u riixdo ilaa dariiqa hoos si degdeg ah-state disks. Waxaa caan ku ah DeepSpeed's ZeRO-Infinity iyo ZeRO-Offload, farsamadu waxay ka baayacmushtareysaa xawaaraha ceyriinka ah ee awoodda, taasoo u oggolaanaysa hal GPU ama koox yar oo qaab-qaabeeya oo leh balaayiin cabbirro ah.
Aragtida Farsamada
Furaha ayaa ah isku dhafka xogta dhaqdhaqaaqa xisaabinta. Dawladaha Optimizer waxay ku fadhiyaan CPU/NVMe; inta lagu guda jiro ka gudubka gadaale, qaybaha waxaa lagu hormariyaa PCIe wax yar ka hor inta aan loo baahnayn iyo tallaabada hagaajinta lafteeda ayaa inta badan ku shaqeysa CPU. ZeRO-Offload waxay ku haysaa float32 miisaanka sayidkiisa iyo daqiiqadaha Adam ee CPU, sidaa darteed xisaabta hore iyo gadaalba waxay ku sii jirtaa GPU-da. NVMe waxay ku darsataa kayd heersare ah sidaa darteed gobolada cabirka terabyte-ku ku daataan diskka halka qaybaha kulul ay ku jiraan RAM.
Saamaynta Istiraatijiyadeed
Qiimaha iyo miisaaniyada
Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
Go'aamo cad
Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
Xakamaynta tayada
Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
Mustaqbalka Wax-ka-qabashada Gobolka ee CPU iyo NVMe
Sida moodellada ay uga sii kordhayaan xusuusta GPU-da, soo-dejintu waxay noqonaysaa heer aan caadi ahayn. Filo is dhexgalka adag oo leh isku xirnaanta dhaqsaha badan sida NVLink-C2C iyo barkadaha xusuusta ee CXL ee xumeeya soohdinta CPU-GPU, oo lagu daray jadwalayaal caqli badan oo saadaaliya gobolada la hor marinayo. Nashqada xusuusta midaysan sida Grace Hopper waxay yaraynaysaa ciqaabta PCIe, qaab-dhismeedyaduna waxay riixayaan samaynta soo dejinta heerarka kala duwan ee ku dhawaad daah-furnaanta si ay hiwaayadaha u hagaajiyaan moodooyinka waaweyn ee qalabka suubban.
Dhaqangelinta Adduunka-dhabta ah
Hagaajinta 13-bilyan-beeg LLM ah hal macaamiil 24 GB GPU iyadoo la adeegsanayo DeepSpeed Zero-Offload si loogu riixo gobollada Adam CPU RAM.
Shaybaar yar oo cilmi baaris ah oo ku tababaraya moodeel balaayiin-beeg ah oo dhowr GPU-yo ah iyadoo u daadisay dowladaha wax-qabad ee NVMe ee wata ZeRO-Infinity.
Isku-duubnida Wajiga Dedeji habaynta awood u siinaysa soo dejinta CPU si ay isticmaalayaashu u socodsiiyaan shaqooyin hagaagsan oo dhammaystiran oo haddii kale tuuraya khaladaadka xusuusta ka baxsan.
Bilawga qiimaha leh ee kiraysanaya ka jaban, xusuusta hoose ee GPU-yada iyo ku rarista NVMe halkii ay ka bixin lahaayeen kaararka heerka-sare ee 80 GB.
Khatarta & Dariiqyada Ilaalada
Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Qorshe Hawleedka Dhaqangelinta
Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Optimizer State Offloading to CPU and NVMe quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
Aadam iyo Hagaajinta La Qabsiga
Su'aalaha soo noqnoqda
What is Optimizer State Offloading to CPU and NVMe?
Khiyaamo kaydinta xusuusta oo dhigata kaydinta culus ee tababarka (gobolada hagaajinta, gradients, mararka qaarkood miisaan) CPU RAM ama NVMe SSDs halkii ay ka ahaan lahaayeen xusuusta GPU-da yar. Waxay u ogolaataa dadka inay tababaraan moodallo aad uga weyn kuwa xusuusta GPU-da ay si kale u oggolaan karto.
Waa maxay himilada koowaad ee u dajinta dawladaha kor u qaadaya CPU ama NVMe?
Soo dejintu waxay u dhaqaajisaa gobolada wanaajiyaha culus ee GPU-ga CPU RAM ama NVMe, iyada oo xoraynaysa xusuusta GPU si moodooyinka waaweyn loogu tababaro isla qalab isku mid ah.
Loogu talagalay hagaajinta Adam ee isku dhafan, xogta sida caadiga ah u isticmaasha xusuusta UGU BADAN halkiibeeg kasta?
Aadam waxa uu kaydiyaa xawli, kala duwanaansho, iyo miisaan sayid oo sax ah, wadar ahaan 16 bytes halkiibeeggiiba, in ka badan 2-byte miisaanka saxda ah ee saxda ah.
Qaabkee ayaa sifeynaya soo dejin heersare ah oo caan baxay?
DeepSpeed's ZeRO-Offload iyo ZeRO-Infinity waxay soo bandhigtay oo caan ka dhigtay kor u qaadida gobolada CPU iyo NVMe cabir ahaan.
Waa maxay qiimaha waxqabadka ugu weyn ee u dejinta CPU ama NVMe?
Ka guurista gobolada ka baxsan GPU waxay la macno tahay ku wareejinta xogta PCIe ama NVMe, taas oo ka gaabinaysa xusuusta GPU-da, markaa wax soo saarku wuu yaraadaa ilaa lagu daro xisaabinta.
Sidee bay nidaamyada rarista u qariyaan inta badan daahitaanka wareejinta?
Jadwaleeyayaashu waxay ka sii hormariyaan qaybaha loo baahan yahay PCIe halka GPUgu wali xisaabinayo, isgaadhsiinta isdhaafsan ee xisaabinta si loo daboolo daahitaanka.