Incamake
Itumanaho rusange nuburyo itsinda rya GPU rihanahana kandi rigahuza amakuru, kandi NCCL ni isomero rya NVIDIA rituma ibyo guhana byihuta. Ibikorwa nka byose-bigabanya ni umutima wumutima wagabanijwe, guhuza gradients muri buri GPU buri ntambwe.
Itumanaho rusange hamwe na NCCL ni tekinoroji yubaka tekinike igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo.
Kwibira cyane
Guhugura icyitegererezo kinini bivuze ko buri GPU ibara gradients ku gice cyayo bwite, noneho GPUs zose zigomba kumvikana kubisubizo hamwe mbere yintambwe ikurikira. Ihuzabikorwa rikorwa hamwe nibikorwa rusange: kugabanya amafaranga yose muri GPUs kandi bigaha buri wese ibisubizo; byose-gukusanya byegeranya buri gice cya GPU muri kopi yuzuye kuri bose; gutangaza byohereza amakuru ya GPU kubandi; kugabanya-gutatanya ikomatanya noneho igacamo ibice. NCCL (Isomero rya NVIDIA Isomero ry'itumanaho) rishyira mubikorwa neza muri GPUs muri seriveri no kuri seriveri, ukoresheje algorithm ya topologiya-imizi nka mpeta n'ibiti byose-bigabanya. Ikoresha NVLink imbere muri node na InfiniBand cyangwa RoCE hagati yumutwe, kandi niyo nkingi yitumanaho munsi ya PyTorch DDP, FSDP, DeepSpeed, na Megatron.
Ubushishozi
Impeta yose-igabanya ni algorithm ya classique: GPUs ikora impeta yumvikana, kandi amakuru agabanyijemo uduce tuzenguruka kuburyo buri ntambwe ihuza itumanaho, bigatuma ihererekanyabubasha ryuzuye-ryiza kandi hafi ya GPU kubara. Kuri node nyinshi, ibiti bishingiye kuri algorithms bigabanya ubukererwe muguhuza ibisubizo muburyo bukurikira. NCCL auto-itahura topologiya, itora algorithm nziza, kandi irashobora gukuramo imibare yo kugabanya mumurongo hamwe na NVIDIA SHARP, igabanya kabiri amakuru agomba kunyura kumurongo.
Kumenya Itumanaho rusange hamwe na NCCL
Itumanaho rusange nuburyo itsinda rya GPU rihanahana kandi rigahuza amakuru, kandi NCCL ni isomero rya NVIDIA rituma ibyo guhana byihuta. Ibikorwa nka byose-bigabanya ni umutima wumutima wagabanijwe, guhuza gradients muri buri GPU buri ntambwe. Itumanaho rusange hamwe na NCCL ni tekinoroji yubaka tekinike igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo. Kugirango wubake byimbitse, fata Itumanaho rusange hamwe na NCCL nkicyitegererezo gikora, ntabwo ari ikintu kimwe: gusobanura ibyagezweho, gusobanura ibitekerezo, no gutandukanya ibyo sisitemu ishobora gukora byizewe nibisaba guca imanza zinzobere.
Mu myitozo, amatsinda akomeye akoresha Itumanaho hamwe na NCCL atezimbere ubwubatsi, amakuru, hamwe nibikorwa remezo birwanya kwizerwa nigiciro. Bandika ibipimo ngenderwaho byerekana intsinzi, bagerageza kurwanya amakuru afatika hamwe nakazi keza, kandi bagasubiramo bashingiye kubikorwa byagaragaye ko batsinzwe aho gutsinda inshuro imwe. Aha niho imyumvire yubumenyi ihinduka mubushobozi burambye kubicuruzwa, politiki, nibikorwa.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mugihe kimwe, Guhindura igipimo kimwe gishobora guhisha intege nke za sisitemu. Uburyo bukomeye cyane ni uguhuza umuvuduko wikigereranyo hamwe na disipuline yimiyoborere: kuyobora abaderevu, gufata ibimenyetso, gutangaza ibyemezo byicyemezo, no gukomeza kuvugurura uburyo bwo kwirinda nkimyitwarire yicyitegererezo, ibyo abakoresha bategereje, nibisabwa n'amategeko bigenda bihinduka.
Ingaruka z'Ingamba
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Gushyira mu bikorwa Isi
Guhuza gradients buri ntambwe yo guhugura muri GPU zose ukoresheje byose-bigabanya muri PyTorch IkwirakwizwaDataParallel
Gusangira optimizer ivuga no gukusanya ibipimo kubisabwa hamwe-byose hamwe no kugabanya-gutatanya muri FSDP cyangwa DeepSpeed ZeRO
Kwamamaza ibipimo byambere byerekana uburemere kuva GPU imwe kubandi bose mugitangira imyitozo
Gukoresha impeta byose-bigabanye hejuru ya NVLink na InfiniBand kugirango ukomeze umurongo mwinshi hejuru ya cluster ya GPU myinshi.
Uburyo bwo Gushyira mu bikorwa
Itumanaho rusange hamwe na NCCL mubikorwa
Guhuza gradients buri ntambwe yo guhugura muri GPU zose ukoresheje byose-bigabanya muri PyTorch IkwirakwizwaDataParallel.
Guhuza gradients buri ntambwe yo guhugura muri GPUs zose ukoresheje kugabanya-kugabanya muri PyTorch IkwirakwizwaDataParallel Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Itumanaho rusange hamwe na NCCL mubikorwa
Gusangira optimizer ivuga no gukusanya ibipimo kubisabwa hamwe-byose hamwe no kugabanya-gutatanya muri FSDP cyangwa DeepSpeed ZeRO.
Sharding optimizer ivuga no gukusanya ibipimo kubisabwa hamwe-byose hamwe no kugabanya-gutatanya muri FSDP cyangwa DeepSpeed ZeRO Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Itumanaho rusange hamwe na NCCL mubikorwa
Kwamamaza ibipimo byambere byerekana uburemere kuva GPU imwe kubandi bose mugitangira imyitozo.
Kwamamaza ibipimo byambere byerekana kuva kuri GPU kugeza kubandi bose mugitangira imyitozo yo kuyobora Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo byimbere, kandi agakurikirana inyungu zibyara umusaruro nibiciro byigihe.
Itumanaho rusange hamwe na NCCL mubikorwa
Gukoresha impeta byose-kugabanya hejuru ya NVLink na InfiniBand kugirango ukomeze umurongo mwinshi hejuru ya classe ya GPU.
Gukoresha impeta byose-bigabanya hejuru ya NVLink na InfiniBand kugirango ukomeze kwaguka hejuru murwego rwinshi rwa GPU Amatsinda ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.