I-Optimizer State Offloading ku-CPU ne-NVMe
Iqhinga lokulondoloza inkumbulo elimisa ukugcinwa kwamabhuku okusindayo kokuqeqeshwa (izimo ze-optimizer, ama-gradient, kwesinye isikhathi izisindo) ku-CPU RAM noma kuma-NVMe SSD esikhundleni sememori ye-GPU eyindlala.
Uhlolojikelele
It lets people train far larger models than their GPU's memory would otherwise allow.
I-Deep Dive
Uma uqeqesha inethiwekhi ye-neural ngesithuthukisi esifana no-Adam, yonke ipharamitha ithwala umthwalo owengeziwe: izibalo ezimbili ezisebenzayo (umfutho nokuhluka), kanye nekhophi enembayo yesisindo, kanye negradient yayo. Ekuqeqesheni okunembayo okuxubile lokhu kungahlanganisa cishe amabhayithi angu-16 ngepharamitha ngayinye, kufiphaze amabhayithi angu-2 ngesisindo ngokwaso. Ukulayisha kususa lowo mthwalo ku-GPU. I-CPU ikhipha i-stream optimizer ingena ohlelweni olujwayelekile lwe-RAM phezu kwebhasi le-PCIe, kuyilapho ukulayishwa kwe-NVMe kuwaphusha kuze kube phansi kumadiski e-solid-state asheshayo. Idume ngabakwa-DeepSpeed's ZeRO-Infinity kanye ne-ZeRO-Offload, le ndlela ihweba ngesivinini esingahluziwe ukuze ibe namandla, ivumela i-GPU eyodwa noma amamodeli amancane eqoqo acule kahle anezigidigidi zamapharamitha.
I-Technical Insight
Ukhiye oweqa ukunyakaza kwedatha nokubala. Izifunda ze-Optimizer zihlala ku-CPU/NVMe; ngesikhathi sokubuyela emuva, ama-partitions alandwa kuqala nge-PCIe ngaphambi nje kokuba adingeke futhi isinyathelo sokulungisa sona sivame ukusebenza ku-CPU. I-ZeRO-Offload igcina izisindo eziyinhloko ze-float32 kanye nezikhathi zika-Adam ku-CPU, ngakho izibalo eziya phambili neziya emuva kuphela ezihlala ku-GPU. I-NVMe yengeza inqolobane enezigaba ukuze isikali se-terabyte sichitheke kudiski kuyilapho izingxenye ezishisayo zihlala ku-RAM.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Le-Optimizer State Offloading ku-CPU ne-NVMe
Njengoba amamodeli eqhubeka nokukhula inkumbulo ye-GPU, ukulayishwa okuhlanganisiwe kuba okujwayelekile kunokuba kungavamile. Lindela ukuhlanganiswa okuqinile ngokuxhumanisa okusheshayo okufana namachibi enkumbulo e-NVLink-C2C kanye ne-CXL afiphalisa umngcele we-CPU-GPU, kanye nabahleli bezinhlelo abahlakaniphe kakhulu ababikezela ukuthi isiphi isimo okufanele silande. Izakhiwo zenkumbulo ezihlanganisiwe ezifana noGrace Hopper zinciphisa inhlawulo ye-PCIe, futhi izinhlaka ziqhubekela phambili ekwenzeni ukulayishwa kwezigaba eziningi kucishe kucace ukuze abathanda ukuzilibazisa bakwazi ukushuna kahle amamodeli amakhulu kuhadiwe elinesizotha.
Ukuqaliswa Komhlaba Wangempela
Ukushuna kahle i-LLM yepharamitha yebhiliyoni engu-13 ku-GPU eyodwa yomthengi ongu-24 GB kusetshenziswa i-DeepSpeed ZeRO-Offload ukuze iphushele izifunda zika-Adam ku-CPU RAM.
Ilebhu encane yocwaningo iqeqesha imodeli yepharamitha yezigidigidi kuma-GPU ambalwa ngokuchitha izifundazwe ze-optimizer kumadrayivu e-NVMe nge-ZeRO-Infinity.
Ubuso Bokubungaza Sheshisa ukulungiselelwa okunika amandla ukulayishwa kwe-CPU ukuze abasebenzisi bakwazi ukwenza imisebenzi yokuhlela kahle engase ikhiphe amaphutha angaphandle kwenkumbulo.
Iziqalo ezikhathalela izindleko eziqasha ama-GPU amafu ashibhile, anenkumbulo ephansi futhi zilayishwa ku-NVMe enamathiselwe esikhundleni sokukhokhela amakhadi aphezulu angama-80 GB.
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Optimizer State Offloading to CPU and NVMe quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Adam kanye ne-Adaptive Optimizers
Imibuzo evame ukubuzwa
What is Optimizer State Offloading to CPU and NVMe?
Iqhinga lokulondoloza inkumbulo elimisa ukugcinwa kwamabhuku okusindayo kokuqeqeshwa (izimo ze-optimizer, ama-gradient, kwesinye isikhathi izisindo) ku-CPU RAM noma kuma-NVMe SSD esikhundleni sememori ye-GPU eyindlala. Ivumela abantu ukuba baqeqeshe amamodeli amakhudlwana kunenkumbulo ye-GPU yabo.
Uyini umgomo oyinhloko wokulayisha izifunda ze-optimizer ku-CPU noma i-NVMe?
Ukulayisha kuhambisa isilungiseleli esisindayo sisuka ku-GPU siye ku-CPU RAM noma i-NVMe, kukhulula inkumbulo ye-GPU ukuze amamodeli amakhulu akwazi ukuqeqeshwa kuhadiwe efanayo.
Okwesilungiseleli sika-Adamu ngokunemba okuxubile, iyiphi idatha evame ukusebenzisa inkumbulo ENGINYANGAYO ngepharamitha ngayinye?
U-Adam ugcina umfutho, ukuhluka, kanye nesisindo esiphelele esinembayo esiyinhloko, esihlanganisa cishe amabhayithi angu-16 ngepharamitha ngayinye, okungaphezulu kakhulu kwesisindo sokunemba esingu-2-byte sesigamu.
Iluphi uhlaka olufaka ukulayishwa kwe-optimizer ngezinga elikhulu?
I-ZeRO-Offload ye-DeepSpeed kanye ne-ZeRO-Infinity yethule futhi yaduma isithuthukisi sokulayisha ku-CPU ne-NVMe esikalini.
Yiziphi izindleko zokusebenza eziphambili zokulayisha ku-CPU noma ku-NVMe?
Izifunda ezihamba ngaphandle kwe-GPU kusho ukudlulisa idatha nge-PCIe noma ku-NVMe, ehamba kancane kunememori ye-GPU, ngakho-ke ukuphuma kwe-output kuyehla ngaphandle uma kugqitshwe ngekhompyutha.
Ingabe amasistimu okulayisha afihla kanjani ukubambezeleka kokudlulisa okuningi?
Abashejuli balanda kuqala ama-partitions adingekayo phezu kwe-PCIe ngenkathi i-GPU isasebenzisa ikhompuyutha, ukuxhumana okudlulanayo nokubala ukubambezeleka kwemaski.