Zazzagewar Jihar Optimizer zuwa CPU da NVMe
Dabarar ceton ƙwaƙwalwar ajiya wanda ke yin fakin ɗaukar nauyi na horo (jahohin ingantawa, gradients, wani lokacin ma'auni) a cikin CPU RAM ko akan NVMe SSDs maimakon ƙarancin ƙwaƙwalwar GPU.
Dubawa
It lets people train far larger models than their GPU's memory would otherwise allow.
Zurfafa nutsewa
Lokacin da kuke horar da hanyar sadarwa ta jijiyoyi tare da ingantawa kamar Adam, kowane siga yana ɗaukar ƙarin kaya: ƙididdiga masu gudana guda biyu (lokaci da bambance-bambancen), da cikakkiyar kwafin nauyi, da gradient. A cikin horarwar daidaitaccen gauraye wannan na iya jimla kusan bytes 16 a kowace siga, yana ɗaukar bytes 2 don nauyin kansa. Sauke kaya yana motsa wannan kaya daga GPU. CPU offload rafi ingantawa jihohin zuwa cikin talakawa tsarin RAM a kan PCIe bas, yayin da NVMe offload tura su zuwa ga m-jihar faifai. Shahararru ta DeepSpeed's ZeRO-Infinity da ZeRO-Offload, dabarar tana cinikin saurin gudu don iya aiki, tana barin GPU guda ɗaya ko ƙaramin gungu mai kyau-tune samfura tare da biliyoyin sigogi.
Fahimtar Fasaha
Makullin shine haɗuwa da motsin bayanai tare da ƙididdigewa. Jihohin ingantawa suna zaune a cikin CPU/NVMe; a lokacin wucewar baya, ana prefetched ɓangarorin akan PCIe kafin a buƙaci su kuma matakin ingantawa da kansa yakan yi aiki akan CPU. ZeRO-Offload yana kiyaye ma'aunin nauyi na float32 da lokacin Adam akan CPU, don haka lissafi na gaba da baya kawai ya tsaya akan GPU. NVMe yana ƙara ma'ajin ƙira don haka jihohin terabyte-sikelin zube zuwa faifai yayin da ɓangarorin zafi ke tsayawa a cikin RAM.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Buɗewar Jiha Mai Ingantawa zuwa CPU da NVMe
Kamar yadda samfura ke ci gaba da haɓaka ƙwaƙwalwar GPU, ƙaddamar da ƙaddamarwa yana zama daidaitattun maimakon m. Yi tsammanin haɗin kai mai ƙarfi tare da haɗin kai cikin sauri kamar NVLink-C2C da wuraren waƙoƙin ƙwaƙwalwar ajiya na CXL waɗanda ke ɓata iyakar CPU-GPU, da ƙwararrun masu tsara jadawalin waɗanda ke hasashen waɗanne jihohin da za a fara. Haɗin gine-ginen ƙwaƙwalwar ajiya irin su Grace Hopper yana rage hukuncin PCIe, kuma ginshiƙai suna matsawa don sanya kaya mai yawa kusan bayyananne don haka masu sha'awar sha'awa za su iya daidaita manyan samfura akan kayan aiki masu sauƙi.
Aiwatar da Gaskiyar Duniya
Kyakkyawan daidaita ma'aunin LLM-biliyan 13 akan GPU mabukaci guda 24 GB ta amfani da DeepSpeed Zero-Offload don tura jihohin Adam zuwa CPU RAM.
Wani ƙaramin ɗakin binciken bincike yana horar da ƙirar siga-biliyan-biyu akan ƴan GPUs ta hanyar zubda jihohin ingantawa zuwa NVMe tuki tare da ZeRO-Infinity.
Rungumar Fuskar Haɓaka saiti waɗanda ke ba da damar saukar da CPU ta yadda masu amfani za su iya gudanar da cikakkun ayyukan daidaitawa waɗanda in ba haka ba za su jefa kurakuran ƙwaƙwalwar ajiya.
Farawa masu ƙima suna yin hayar mai rahusa, GPUs masu ƙarancin ƙwaƙwalwar ajiya da saukarwa zuwa haɗe-haɗe na NVMe maimakon biyan manyan katunan 80 GB.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Optimizer State Offloading to CPU and NVMe quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Adam and Adaptive Optimizers
Tambayoyin da ake yawan yi
What is Optimizer State Offloading to CPU and NVMe?
Dabarar ceton ƙwaƙwalwar ajiya wanda ke yin fakin ɗaukar nauyi na horo (jahohin ingantawa, gradients, wani lokacin ma'auni) a cikin CPU RAM ko akan NVMe SSDs maimakon ƙarancin ƙwaƙwalwar GPU. Yana ba mutane damar horar da ƙira mafi girma fiye da ƙwaƙwalwar GPU ɗin su in ba haka ba.
Menene babban burin sauke jihohin ingantawa zuwa CPU ko NVMe?
Zazzagewa yana motsa jahohin ingantawa masu nauyi daga GPU zuwa CPU RAM ko NVMe, yana 'yantar da ƙwaƙwalwar GPU don haka za'a iya horar da manyan samfura akan kayan aiki iri ɗaya.
Ga na'urar inganta Adam a cikin gauraye daidaitattun daidaito, wanne bayanai yawanci ke cinye MAFI KYAU ƙwaƙwalwar ajiya a kowace siga?
Adam yana adana kuzari, bambance-bambance, da cikakken madaidaicin ma'aunin nauyi, jimlar kusan bytes 16 a kowace siga, fiye da madaidaicin rabin-byte na rabin-byte.
Wanne tsarin tsarin da ya shahara da manyan sikelin ingantawa?
DeepSpeed's ZeRO-Offload da ZeRO-Infinity an gabatar da su da kuma haɓaka jihohin haɓaka haɓakawa zuwa CPU da NVMe a sikelin.
Menene babban farashin aikin fitarwa zuwa CPU ko NVMe?
Matsar da jihohi a kashe-GPU yana nufin canja wurin bayanai akan PCIe ko zuwa NVMe, wanda yayi hankali fiye da ƙwaƙwalwar GPU akan GPU, don haka abin da ake sarrafawa yana raguwa sai dai idan an haɗa shi da lissafi.
Ta yaya tsarin saukewa ke ɓoye yawancin jinkirin canja wuri?
Masu tsara tsarawa suna ƙaddamar da ɓangarorin da ake buƙata akan PCIe yayin da GPU ke ci gaba da ƙididdigewa, sadarwa tare da ƙididdigewa don rufe latency.