Itumanaho rusange hamwe na NCCL
Itumanaho rusange nuburyo itsinda rya GPU rihanahana kandi rigahuza amakuru, kandi NCCL ni isomero rya NVIDIA rituma ibyo guhana byihuta.
Incamake
Operations like all-reduce are the heartbeat of distributed training, synchronizing gradients across every GPU each step.
Kwibira cyane
Guhugura icyitegererezo kinini bivuze ko buri GPU ibara gradients ku gice cyayo bwite, noneho GPUs zose zigomba kumvikana kubisubizo hamwe mbere yintambwe ikurikira. Ihuzabikorwa rikorwa hamwe nibikorwa rusange: kugabanya amafaranga yose muri GPUs kandi bigaha buri wese ibisubizo; byose-gukusanya byegeranya buri gice cya GPU muri kopi yuzuye kuri bose; isakazabutumwa ryohereza amakuru ya GPU ahasigaye; kugabanya-gutatanya ikomatanya noneho igacamo ibice. NCCL (Isomero rya NVIDIA Isomero ry'itumanaho) rishyira mubikorwa neza muri GPUs muri seriveri no kuri seriveri, ukoresheje algorithm ya topologiya-imizi nka mpeta n'ibiti byose-bigabanya. Ikoresha NVLink imbere muri node na InfiniBand cyangwa RoCE hagati yumutwe, kandi niyo nkingi yitumanaho munsi ya PyTorch DDP, FSDP, DeepSpeed, na Megatron.
Ubushishozi
Impeta yose-igabanya ni algorithm ya classique: GPUs ikora impeta yumvikana, kandi amakuru agabanyijemo uduce tuzenguruka kuburyo buri ntambwe ihuza itumanaho, bigatuma ihererekanyabubasha ryuzuye-ryiza kandi hafi ya GPU kubara. Kuri node nyinshi, ibiti bishingiye kuri algorithms bigabanya ubukererwe muguhuza ibisubizo muburyo bukurikira. NCCL auto-itahura topologiya, itora algorithm nziza, kandi irashobora gukuramo imibare yo kugabanya mumurongo hamwe na NVIDIA SHARP, igabanya kabiri amakuru agomba kunyura kumurongo.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza h'itumanaho rusange hamwe na NCCL
Nka cluster igera ku bihumbi magana GPUs, itumanaho rigenda ryiganje mugihe cyamahugurwa, nuko amasomero rusange arumupaka ushyushye. Tegereza byimbitse muri neti ya comptabilite (switch ikora kugabanya), guhuzagurika neza kubara no gutumanaho kugirango uhishe ubukererwe, hamwe na hamwe-hamwe-hamwe-hamwe-hamwe-hamwe-hamwe-na-hamwe-na-hamwe-na-hamwe-na-hamwe-na-hamwe-na-hamwe-na-hamwe-bigabanya kugabanuka bytes yimutse. Irushanwa naryo riragenda ryiyongera, hamwe nimbaraga zambukiranya ibicuruzwa hamwe na Ethernet ishingiye kuri RDMA isunika ubundi buryo, mugihe NCCL ikomeje gukaza umurego hamwe na NVLink, NVSwitch, hamwe nigitambara cyiza cya optique.
Gushyira mu bikorwa Isi
Guhuza gradients buri ntambwe yo guhugura muri GPU zose ukoresheje byose-bigabanya muri PyTorch IkwirakwizwaDataParallel
Gusangira optimizer ivuga no gukusanya ibipimo kubisabwa hamwe-byose hamwe no kugabanya-gutatanya muri FSDP cyangwa DeepSpeed ZeRO
Kwamamaza ibipimo byambere byerekana uburemere kuva GPU imwe kubandi bose mugitangira imyitozo
Gukoresha impeta byose-bigabanye hejuru ya NVLink na InfiniBand kugirango ukomeze umurongo mwinshi hejuru ya cluster ya GPU myinshi.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Collective Communication and NCCL quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Kurubuga na Offline Ikiranga Gukora Skew
Ibibazo bikunze kubazwa
What is Collective Communication and NCCL?
Itumanaho rusange nuburyo itsinda rya GPU rihanahana kandi rigahuza amakuru, kandi NCCL ni isomero rya NVIDIA rituma ibyo guhana byihuta. Ibikorwa nka byose-bigabanya ni umutima wumutima wagabanijwe, guhuza gradients muri buri GPU buri ntambwe.
Niki ibikorwa byose bigabanya gukora mumahugurwa yatanzwe?
Byose-bigabanye amafaranga (cyangwa ubundi ikomatanya) agaciro muri buri GPU kandi ikwirakwiza ibisubizo bimwe bigaruka kuri bose, nuburyo gradients ihuzwa.
Amagambo ahinnye NCCL agereranya iki?
NCCL ni isomero rya NVIDIA Isomero rusange ryitumanaho, rishyira mubikorwa byihuse byinshi-GPU hamwe nibikorwa byinshi.
Ni ukubera iki impeta zose zigabanywa zifatwa nk'umugozi mwiza?
Mugukata amakuru no gutambutsa ibice bizengurutse impeta yumvikana, buri murongo ukoreshwa icyarimwe kandi ihererekanyabubasha ryose ryigenga hafi yumubare wa GPUs.
Nibihe bikorwa rusange bikusanya buri GPU yamakuru muri kopi yuzuye ifitwe na GPU zose?
Byose-byegeranya bikusanya ibice bitandukanye muri buri GPU kandi bigateranya ibyuzuye kuri byose, bikoreshwa cyane mumyitozo ikarishye nka FSDP.
Niki NVIDIA SHARP ikora mubikorwa rusange?
SHARP ikora muri comptabilite, ikora igice cyo kugabanya imbere muri switch kugirango amakuru make agomba kunyura kumurongo, kwihuta hamwe.