Gukwirakwiza Leta Kurekura kuri CPU na NVMe
Amayeri yo kuzigama yibitse ahagarika kubika ibitabo biremereye byamahugurwa (optimizer leta, gradients, rimwe na rimwe uburemere) muri CPU RAM cyangwa kuri NVMe SSDs aho kuba GPU yibuke.
Incamake
It lets people train far larger models than their GPU's memory would otherwise allow.
Kwibira cyane
Iyo uhuguye urusobe rw'imitsi hamwe na optimizer nka Adam, buri kintu cyose gitwara imizigo yinyongera: imibare ibiri ikora (umuvuduko na variance), wongeyeho kopi yuzuye yuburemere, wongeyeho na gradient yayo. Mu mahugurwa avanze-yuzuye neza ibi birashobora kuba hafi ya 16 bytes kuri parameter, bikagabanya 2 bytes kuburemere ubwabwo. Kurekura bimura iyo mitwaro kuri GPU. CPU yamanura imitekerereze ya optimizer ivuga muri sisitemu isanzwe RAM hejuru ya bisi ya PCIe, mugihe NVMe yamanuye ibasunika kugeza kumanuka yihuta ya disiki ikomeye. Yamamaye cyane muri ZeRO-Infinity na ZeRO-Offload ya DeepSpeed, tekinike igurisha umuvuduko muke kubushobozi, ikareka GPU imwe cyangwa cluster ntoya ihuza neza na miliyari zingana.
Ubushishozi
Urufunguzo ni uguhuza amakuru yimibare hamwe no kubara. Leta optimizer yicara muri CPU / NVMe; mugihe cyinyuma cyinyuma, ibice byateguwe hejuru ya PCIe mbere yuko bikenerwa kandi intambwe ya optimizer ubwayo ikora kuri CPU. ZeRO-Offload ituma uburemere bwa float32 hamwe nigihe cya Adamu kuri CPU, gusa imibare yimbere ninyuma iguma kuri GPU. NVMe yongeyeho cache itondekanye kuburyo terabyte-nini ya leta isuka kuri disiki mugihe ibice bishyushye biguma muri RAM.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Kazoza ka Optimizer Leta Yipakurura kuri CPU na NVMe
Nka moderi ikomeza kwiyongera kububiko bwa GPU, gupakurura ibyiciro bigenda biba ibisanzwe aho kuba exotic. Tegereza guhuza cyane hamwe n’imikoranire yihuse nka NVLink-C2C na CXL yibuka yibidendezi bihindura imipaka ya CPU-GPU, wongeyeho gahunda zubwenge ziteganya ibizaba mbere yo gutangira. Ububiko-bwibumbiye hamwe nka Grace Hopper bugabanya ibihano bya PCIe, kandi urwego rugenda rutera gukora ibintu byinshi byo gukuramo ibintu hafi ya mucyo kugirango abakunda ibintu bashobore guhuza neza moderi nini ku byuma byoroheje.
Gushyira mu bikorwa Isi
Kuringaniza neza miliyari 13-ibipimo bya LLM kuri GPU imwe ya 24 GB ikoresha GPS ukoresheje DeepSpeed ZeRO-Offload kugirango usunike leta za Adam RAM RAM.
Laboratoire ntoya yubushakashatsi ihugura miliyari-nyinshi-yerekana urugero kuri GPU nkeya isuka leta nziza kuri drives ya NVMe hamwe na ZeRO-Infinity.
Guhobera Isura Kwihutisha ibice bifasha CPU gukuramo kugirango abakoresha bashobore gukora imirimo yuzuye neza-itondagura ubundi buryo bwo guta amakosa yibuka.
Gutangiza ibiciro bikodesha bihendutse, byibuke-yibuke rya GPUs no gupakurura kuri NVMe yometse aho kwishyura amakarita yo mu rwego rwo hejuru 80 GB.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Optimizer State Offloading to CPU and NVMe quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Adam na Adaptive Optimizers
Ibibazo bikunze kubazwa
What is Optimizer State Offloading to CPU and NVMe?
Amayeri yo kuzigama yibitse ahagarika kubika ibitabo biremereye byamahugurwa (optimizer leta, gradients, rimwe na rimwe uburemere) muri CPU RAM cyangwa kuri NVMe SSDs aho kuba GPU yibuke. Bituma abantu bitoza moderi nini cyane kuruta ububiko bwa GPU bwakwemerera.
Niyihe ntego yibanze yo gupakurura optimizer leta kuri CPU cyangwa NVMe?
Kurekura byimura optimizer iremereye kuri GPU muri RAM RAM cyangwa NVMe, kubohora ububiko bwa GPU kuburyo moderi nini zishobora gutozwa ibyuma bimwe.
Kubantu ba optimizateri muburyo buvanze neza, ni ayahe makuru asanzwe akoresha ububiko bwinshi kuri buri kintu?
Adamu abika imbaraga, itandukaniro, hamwe nuburemere bwuzuye bwuzuye, byose hamwe bigera kuri 16 byite kuri parameter, birenze kure uburemere bwa 2-byte.
Nibihe bikoresho biranga abantu benshi bakwirakwiza optimizer?
DeepSpeed's ZeRO-Offload na ZeRO-Infinity yatangije kandi ikwirakwiza ibimanuka byo gukuramo ibintu kuri CPU na NVMe kurwego.
Ni ikihe giciro nyamukuru cyibikorwa byo gupakurura kuri CPU cyangwa NVMe?
Kwimura leta hanze ya GPU bisobanura kohereza amakuru hejuru ya PCIe cyangwa kuri NVMe, itinda kurenza ububiko bwa GPU, bityo ibicuruzwa byinjira bikamanuka keretse byuzuye na compte.
Nigute sisitemu yo gupakurura ihisha byinshi mubukererwe?
Abategura bategura ibice bikenewe hejuru ya PCIe mugihe GPU ikiri kubara, guhuza itumanaho hamwe no kubara kubitinda bya mask.