Incamake
Amayeri yo kuzigama yibitse ahagarika kubika ibitabo biremereye byamahugurwa (optimizer leta, gradients, rimwe na rimwe uburemere) muri CPU RAM cyangwa kuri NVMe SSDs aho kuba GPU yibuke. Bituma abantu bitoza moderi nini cyane kuruta ububiko bwa GPU bwakwemerera.
Optimizer State Offloading to CPU na NVMe nikibanza cyubaka tekinike kigira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo.
Kwibira cyane
Iyo uhuguye urusobe rw'imitsi hamwe na optimizer nka Adam, buri kintu cyose gitwara imizigo yinyongera: imibare ibiri ikora (umuvuduko na variance), wongeyeho kopi yuzuye yuburemere, wongeyeho na gradient yayo. Mu mahugurwa avanze-yuzuye neza ibi birashobora kuba hafi ya 16 bytes kuri parameter, bikagabanya 2 bytes kuburemere ubwabwo. Kurekura bimura iyo mitwaro kuri GPU. CPU yamanura imitekerereze ya optimizer ivuga muri sisitemu isanzwe RAM hejuru ya bisi ya PCIe, mugihe NVMe yamanuye ibasunika kugeza kumanuka yihuta ya disiki ikomeye. Yamamaye cyane muri ZeRO-Infinity na ZeRO-Offload ya DeepSpeed, tekinike igurisha umuvuduko muke kubushobozi, ikareka GPU imwe cyangwa cluster ntoya ihuza neza na miliyari zingana.
Ubushishozi
Urufunguzo ni uguhuza amakuru yimibare hamwe no kubara. Leta optimizer yicara muri CPU / NVMe; mugihe cyinyuma cyinyuma, ibice byateguwe hejuru ya PCIe mbere yuko bikenerwa kandi intambwe ya optimizer ubwayo ikora kuri CPU. ZeRO-Offload ituma uburemere bwa float32 hamwe nigihe cya Adamu kuri CPU, gusa imibare yimbere ninyuma iguma kuri GPU. NVMe yongeyeho cache itondekanye kuburyo terabyte-nini ya leta isuka kuri disiki mugihe ibice bishyushye biguma muri RAM.
Kumenya Optimizer Leta Kurekura CPU na NVMe
Amayeri yo kubika kwibuka ahagarika kubika ibitabo biremereye byamahugurwa (leta ya optimizer, gradients, rimwe na rimwe uburemere) muri RAM RAM cyangwa kuri NVMe SSDs aho kuba GPU yibuke. Bituma abantu bitoza moderi nini cyane kuruta ububiko bwa GPU bwakwemerera. Optimizer State Offloading to CPU na NVMe nikibanza cyubaka tekinike kigira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo. Kugirango wubake byimbitse, fata Optimizer State Offloading kuri CPU na NVMe nkicyitegererezo gikora, ntabwo ari ikintu kimwe: gusobanura ibyagezweho, gusobanura ibitekerezo, no gutandukanya ibyo sisitemu ishobora gukora byizewe nibisaba guca imanza zinzobere.
Mubimenyerezo, amakipe akomeye akoresha Optimizer ya Leta Yipakurura kuri CPU na NVMe atezimbere imyubakire, amakuru, hamwe nibikorwa remezo birwanya kwizerwa nigiciro. Bandika ibipimo ngenderwaho byerekana intsinzi, bagerageza kurwanya amakuru afatika hamwe nakazi keza, kandi bagasubiramo bashingiye kubikorwa byagaragaye ko batsinzwe aho gutsinda inshuro imwe. Aha niho imyumvire yubumenyi ihinduka mubushobozi burambye kubicuruzwa, politiki, nibikorwa.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mugihe kimwe, Guhindura igipimo kimwe gishobora guhisha intege nke za sisitemu. Uburyo bukomeye cyane ni uguhuza umuvuduko wikigereranyo hamwe na disipuline yimiyoborere: kuyobora abaderevu, gufata ibimenyetso, gutangaza ibyemezo byicyemezo, no gukomeza kuvugurura uburyo bwo kwirinda nkimyitwarire yicyitegererezo, ibyo abakoresha bategereje, nibisabwa n'amategeko bigenda bihinduka.
Ingaruka z'Ingamba
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Gushyira mu bikorwa Isi
Kuringaniza neza miliyari 13-ibipimo bya LLM kuri GPU imwe ya 24 GB ikoresha GPS ukoresheje DeepSpeed ZeRO-Offload kugirango usunike leta za Adam RAM RAM.
Laboratoire ntoya yubushakashatsi ihugura miliyari-nyinshi-yerekana urugero kuri GPU nkeya isuka leta nziza kuri drives ya NVMe hamwe na ZeRO-Infinity.
Guhobera Isura Kwihutisha ibice bifasha CPU gukuramo kugirango abakoresha bashobore gukora imirimo yuzuye neza-itondagura ubundi buryo bwo guta amakosa yibuka.
Gutangiza ibiciro bikodesha bihendutse, byibuke-yibuke rya GPUs no gupakurura kuri NVMe yometse aho kwishyura amakarita yo mu rwego rwo hejuru 80 GB.
Uburyo bwo Gushyira mu bikorwa
Gukwirakwiza Leta Kurekura kuri CPU na NVMe mubikorwa
Kuringaniza neza miliyari 13-ibipimo bya LLM kuri GPU imwe ya 24 GB ikoresha GPS ukoresheje DeepSpeed ZeRO-Offload kugirango usunike leta za Adam RAM RAM.
Kuringaniza neza miliyari 13-ibipimo bya LLM kumurongo umwe wa 24 GB ukoresha GPU ukoresheje DeepSpeed ZeRO-Offload kugirango usunike leta za Adam muri CPU RAM Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byimbere imbere, agakomeza inzira yo kuzamura abantu kubibazo byimbitse, kandi akurikirana inyungu zibyara umusaruro nibiciro byigihe.
Gukwirakwiza Leta Kurekura kuri CPU na NVMe mubikorwa
Laboratoire ntoya yubushakashatsi ihugura miliyari-nyinshi-yerekana urugero kuri GPU nkeya isuka leta nziza kuri drives ya NVMe hamwe na ZeRO-Infinity.
Laboratoire ntoya yubushakashatsi ihugura miriyari-miriyari-yerekana urugero kuri GPU nkeya isuka leta nziza kuri disiki ya NVMe hamwe na ZeRO-Infinity Amakipe ubusanzwe ibona umusaruro ushimishije mugihe isobanura ibipimo ngenderwaho byimbere, ikomeza inzira yo kuzamura abantu kubibazo byimbitse, kandi ikurikirana inyungu zibyara umusaruro nibiciro byigihe.
Gukwirakwiza Leta Kurekura kuri CPU na NVMe mubikorwa
Guhobera Isura Kwihutisha ibice bifasha CPU gukuramo kugirango abakoresha bashobore gukora imirimo yuzuye neza-itondagura ubundi buryo bwo guta amakosa yibuka.
Guhobera Isura Kwihutisha porogaramu zituma CPU ikuramo ibintu kugirango abakoresha bashobore gukora imirimo yuzuye-itunganya neza ubundi buryo bwo guta amakosa yibukwa Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo byimbere, kandi akurikirana inyungu zibyara umusaruro nibiciro byigihe.
Gukwirakwiza Leta Kurekura kuri CPU na NVMe mubikorwa
Gutangiza ibiciro bikodesha bihendutse, byibuke-yibuke rya GPUs no gupakurura kuri NVMe yometse aho kwishyura amakarita yo mu rwego rwo hejuru 80 GB.
Ibiciro bitangira gukodesha gukodesha bihendutse, byibuze-yibuka igicu GPUs no gupakurura kuri NVMe aho kwishyurira amakarita yo mu rwego rwo hejuru ya 80 GB Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo bikurikirana, kandi akurikirana inyungu zibyara umusaruro nibiciro byigihe.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.