Dubawa
Dabarar ceton ƙwaƙwalwar ajiya wanda ke yin fakin ɗaukar nauyi na horo (jahohin ingantawa, gradients, wani lokacin ma'auni) a cikin CPU RAM ko akan NVMe SSDs maimakon ƙarancin ƙwaƙwalwar GPU. Yana ba mutane damar horar da ƙira mafi girma fiye da ƙwaƙwalwar GPU ɗin su in ba haka ba.
Ƙaddamar da Jiha Mai Haɓakawa zuwa CPU da NVMe tubalin ginin fasaha ne wanda ke shafar ingancin ƙira, farashin kayan aiki, latency, da aminci a sikeli.
Zurfafa nutsewa
Lokacin da kuke horar da hanyar sadarwa ta jijiyoyi tare da ingantawa kamar Adam, kowane siga yana ɗaukar ƙarin kaya: ƙididdiga masu gudana guda biyu (lokaci da bambance-bambancen), da cikakkiyar kwafin nauyi, da gradient. A cikin horarwar daidaitaccen gauraye wannan na iya jimla kusan bytes 16 a kowace siga, yana ɗaukar bytes 2 don nauyin kansa. Sauke kaya yana motsa wannan kaya daga GPU. CPU offload rafi ingantawa jihohin zuwa cikin talakawa tsarin RAM a kan PCIe bas, yayin da NVMe offload tura su zuwa ga m-jihar faifai. Shahararru ta DeepSpeed's ZeRO-Infinity da ZeRO-Offload, dabarar tana cinikin saurin gudu don iya aiki, tana barin GPU guda ɗaya ko ƙaramin gungu mai kyau-tune samfura tare da biliyoyin sigogi.
Fahimtar Fasaha
Makullin shine haɗuwa da motsin bayanai tare da ƙididdigewa. Jihohin ingantawa suna zaune a cikin CPU/NVMe; a lokacin wucewar baya, ana prefetched ɓangarorin akan PCIe kafin a buƙaci su kuma matakin ingantawa da kansa yakan yi aiki akan CPU. ZeRO-Offload yana kiyaye ma'aunin nauyi na float32 da lokacin Adam akan CPU, don haka lissafi na gaba da baya kawai ya tsaya akan GPU. NVMe yana ƙara ma'ajin ƙira don haka jihohin terabyte-sikelin zube zuwa faifai yayin da ɓangarorin zafi ke tsayawa a cikin RAM.
Mastering Optimizer State Offloading zuwa CPU da NVMe
Dabarar ceton ƙwaƙwalwar ajiya wanda ke yin fakin ɗaukar nauyi na horo (jahohin ingantawa, gradients, wani lokacin ma'auni) a cikin CPU RAM ko akan NVMe SSDs maimakon ƙarancin ƙwaƙwalwar GPU. Yana ba mutane damar horar da ƙira mafi girma fiye da ƙwaƙwalwar GPU ɗin su in ba haka ba. Ƙaddamar da Jiha Mai Haɓakawa zuwa CPU da NVMe tubalin ginin fasaha ne wanda ke shafar ingancin ƙira, farashin kayan aiki, latency, da aminci a sikeli. Don haɓaka fahimta mai zurfi, bi da Ƙaddamarwa na Jiha zuwa CPU da NVMe azaman ƙirar aiki, ba fasali ɗaya ba: ayyana sakamakon da ake so, fayyace zato, da raba abin da tsarin zai iya yi da dogaro daga abin da har yanzu yana buƙatar yanke hukunci na ƙwararru.
A aikace, ƙungiyoyi masu ƙarfi da ke amfani da Haɓakawa na Jiha zuwa CPU da NVMe suna haɓaka gine-gine, bayanai, da zaɓin abubuwan more rayuwa akan dogaro da farashi. Suna rubuta ƙayyadaddun ƙa'idodin nasara, gwaji akan bayanan gaskiya da gudanawar aiki, da jujjuyawar bisa ga tsarin gazawar da aka lura maimakon cin nasara na lokaci ɗaya. Wannan shine inda fahimtar ka'idar ta juya zuwa iyawa mai dorewa a cikin samfura, manufofi, da ayyuka.
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru. A lokaci guda, Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin. Hanyar da ta fi dacewa ita ce haɗa saurin gwaji tare da horon gudanarwa: gudanar da matukin jirgi, kama shaida, buga rajistan ayyukan yanke shawara, da ci gaba da sabunta abubuwan tsaro kamar yadda halayen ƙira, tsammanin mai amfani, da buƙatun tsari ke tasowa.
Dabarun Tasiri
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru. A cikin ƙawance masu inganci, ana fassara wannan zuwa ƙa'idodin aiki waɗanda za a iya aunawa, iyakokin ikon mallaka, da kuma bita-da-kullin bita don ƙungiyoyi su iya haɓaka kwarin gwiwa a maimakon ɓata shakku.
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba. A cikin ƙawance masu inganci, ana fassara wannan zuwa ƙa'idodin aiki waɗanda za a iya aunawa, iyakokin ikon mallaka, da kuma bita-da-kullin bita don ƙungiyoyi su iya haɓaka kwarin gwiwa a maimakon ɓata shakku.
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa. A cikin ƙawance masu inganci, ana fassara wannan zuwa ƙa'idodin aiki waɗanda za a iya aunawa, iyakokin ikon mallaka, da kuma bita-da-kullin bita don ƙungiyoyi su iya haɓaka kwarin gwiwa a maimakon ɓata shakku.
Aiwatar da Gaskiyar Duniya
Kyakkyawan daidaita ma'aunin LLM-biliyan 13 akan GPU mabukaci guda 24 GB ta amfani da DeepSpeed Zero-Offload don tura jihohin Adam zuwa CPU RAM.
Wani ƙaramin ɗakin binciken bincike yana horar da ƙirar siga-biliyan-biyu akan ƴan GPUs ta hanyar zubda jihohin ingantawa zuwa NVMe tuki tare da ZeRO-Infinity.
Rungumar Fuskar Haɓaka saiti waɗanda ke ba da damar saukar da CPU ta yadda masu amfani za su iya gudanar da cikakkun ayyukan daidaitawa waɗanda in ba haka ba za su jefa kurakuran ƙwaƙwalwar ajiya.
Farawa masu ƙima suna yin hayar mai rahusa, GPUs masu ƙarancin ƙwaƙwalwar ajiya da saukarwa zuwa haɗe-haɗe na NVMe maimakon biyan manyan katunan 80 GB.
Hanyoyin Aiwatarwa
Zazzagewar Jiha mai ingantawa zuwa CPU da NVMe a aikace
Kyakkyawan daidaita ma'aunin LLM-biliyan 13 akan GPU mabukaci guda 24 GB ta amfani da DeepSpeed Zero-Offload don tura jihohin Adam zuwa CPU RAM.
Daidaita daidaitaccen sigar 13-biliyan LLM akan mabukaci na 24 GB guda ɗaya ta amfani da DeepSpeed ZeRO-Offload don tura jihohin Adam zuwa CPU RAM Teams yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓaka ɗan adam don ƙararraki, da bin diddigin nasarorin samarwa da ƙimar kuskure akan lokaci.
Zazzagewar Jiha mai ingantawa zuwa CPU da NVMe a aikace
Wani ƙaramin ɗakin binciken bincike yana horar da ƙirar siga-biliyan-biyu akan ƴan GPUs ta hanyar zubda jihohin ingantawa zuwa NVMe tuki tare da ZeRO-Infinity.
Wani ƙaramin ɗakin binciken bincike yana horar da ƙirar ƙirar biliyan-biliyoyin akan ƴan GPUs ta hanyar zubar da jihohin ingantawa zuwa NVMe tare da ƙungiyoyin ZeRO-Infinity yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓakar ɗan adam don ƙararraki, da bin diddigin abubuwan samarwa da ƙimar kuskure akan lokaci.
Zazzagewar Jiha mai ingantawa zuwa CPU da NVMe a aikace
Rungumar Fuskar Haɓaka saiti waɗanda ke ba da damar saukar da CPU ta yadda masu amfani za su iya gudanar da cikakkun ayyukan daidaitawa waɗanda in ba haka ba za su jefa kurakuran ƙwaƙwalwar ajiya.
Hugging Face Haɓaka saiti waɗanda ke ba da damar saukar da CPU don haka masu amfani za su iya gudanar da cikakken ayyukan daidaitawa waɗanda in ba haka ba za su jefa kurakuran ƙwaƙwalwar ajiya Ƙungiyoyi yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓakar ɗan adam don ƙararraki, da bin diddigin nasarorin samarwa da ƙimar kuskure akan lokaci.
Zazzagewar Jiha mai ingantawa zuwa CPU da NVMe a aikace
Farawa masu ƙima suna yin hayar mai rahusa, GPUs masu ƙarancin ƙwaƙwalwar ajiya da saukarwa zuwa haɗe-haɗe na NVMe maimakon biyan manyan katunan 80 GB.
Farawa masu ƙima suna yin hayar mai rahusa, GPUs mai ƙarancin ƙwaƙwalwar ajiya da saukewa zuwa NVMe a haɗe maimakon biyan kuɗi don manyan katunan 80 GB Ƙungiyoyi yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓaka ɗan adam don ƙararraki, da bin diddigin nasarorin samarwa da ƙimar kuskure akan lokaci.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.