Sadarwar Jama'a da NCCL
Sadarwa ta gama gari ita ce yadda rukunin GPUs ke musanya da haɗa bayanai, kuma NCCL ita ce ɗakin karatu na NVIDIA wanda ke sanya waɗancan musayar musayar wuta cikin sauri.
Dubawa
Operations like all-reduce are the heartbeat of distributed training, synchronizing gradients across every GPU each step.
Zurfafa nutsewa
Horar da babban samfuri yana nufin kowane GPU yana lissafta gradients akan nasa yanki na bayanai, sannan duk GPUs dole ne su yarda da sakamakon haɗin gwiwa kafin mataki na gaba. Ana yin wannan haɗin kai tare da ayyukan gama kai: duk-rage ƙimar ƙima a cikin GPUs kuma yana ba kowa sakamako; duk-taron tattara kowane yanki na GPU zuwa cikakken kwafi akan su duka; watsa shirye-shirye yana aika bayanan GPU ɗaya zuwa sauran; rage-watse ya hade sannan ya rabu. NCCL (NVIDIA Collective Communications Library) yana aiwatar da waɗannan da kyau a cikin GPUs a cikin uwar garken da kuma cikin sabar, ta amfani da algorithms na topology-sani kamar zobe da itace duk-rage. Yana amfani da NVLink a cikin kumburi da InfiniBand ko RoCE tsakanin nodes, kuma shine kashin bayan sadarwa a ƙarƙashin PyTorch DDP, FSDP, DeepSpeed, da Megatron.
Fahimtar Fasaha
Ring duk-rage shine al'adar algorithm: GPUs suna samar da zobe na ma'ana, kuma an raba bayanan zuwa gungu-gungu waɗanda ke yawo don haka kowane mataki ya mamaye sadarwa, yana mai da jimlar canja wurin bandwidth-mafi kyau kuma mai zaman kansa daga ƙididdigar GPU. Ga nodes da yawa, algorithms na tushen itace suna rage jinkiri ta hanyar haɗa sakamako cikin matsayi. NCCL auto-gane topology, zabar mafi kyawun algorithm, kuma zai iya saukar da raguwar lissafi a cikin hanyar sadarwa tare da NVIDIA SHARP, ragi bayanan da dole ne su ketare hanyoyin haɗin gwiwa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Sadarwar Sadarwa da NCCL
Kamar yadda gungu ya kai dubun dubatar GPUs, sadarwa tana ƙara mamaye lokacin horo, don haka ɗakunan karatu na gama gari suna da iyaka. Yi tsammanin ƙididdige ƙididdiga a cikin hanyar sadarwa mai zurfi (masu sauya suna yin raguwa), mafi kyawun haɗar lissafi da sadarwa don ɓoye latency, da ƙananan madaidaicin ƙungiyoyi waɗanda ke rage tasirin bytes. Gasar kuma tana haɓaka, tare da ƙoƙarin ƙetare-tsaye da RDMA na tushen Ethernet yana tura madadin, yayin da NCCL ke ci gaba da ƙarfafa haɗin gwiwa tare da NVLink, NVSwitch, da masana'anta masu tasowa.
Aiwatar da Gaskiyar Duniya
Daidaita gradients kowane matakin horo a duk GPUs ta amfani da duk-raguwa a cikin PyTorch DistributedDataParallel
Rarraba jihohin ingantawa da tara sigogi akan buƙatu tare da tattara duka da rage-warwatsawa a cikin FSDP ko DeepSpeed ZeRO
Watsa ma'aunin ƙira na farko daga GPU ɗaya zuwa duk wasu a farkon guduwar horo
Yin amfani da zobe duka-rage akan NVLink da InfiniBand don ci gaba da haɓaka babban bandwidth a cikin gungu na GPU masu yawa.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Collective Communication and NCCL quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Yanar Gizo da Yanar Gizo Yana Hidima Skew
Tambayoyin da ake yawan yi
What is Collective Communication and NCCL?
Sadarwa ta gama gari ita ce yadda rukunin GPUs ke musanya da haɗa bayanai, kuma NCCL ita ce ɗakin karatu na NVIDIA wanda ke sanya waɗancan musayar musayar wuta cikin sauri. Ayyuka kamar duk-raguwa sune bugun zuciya na horarwa da aka rarraba, aiki tare da gradients a kowane GPU kowane mataki.
Menene aikin rage duka ya cim ma a cikin horon da aka rarraba?
Ƙimar Rage Duka (ko in ba haka ba ta haɗa) ƙima a cikin kowane GPU kuma yana rarraba sakamako iri ɗaya ga duka su, wanda shine yadda ake daidaita gradients.
Menene ma'anar gajarta NCCL ta tsaya ga?
NCCL ita ce ɗakin karatu na Sadarwar Sadarwar Sadarwar NVIDIA, wanda ke aiwatar da ayyuka na gama-gari masu yawa da GPU da sauri.
Me yasa ake ɗaukar duk-raguwar zobe-mafi kyawun bandwidth?
Ta hanyar karkatar da bayanai da keɓance yanki a kusa da zobe mai ma'ana, ana amfani da kowane hanyar haɗin gwiwa lokaci guda kuma jimlar canja wuri ta zama mai zaman kanta daga adadin GPUs.
Wane aiki gama gari ne ke tattara kowane yanki na bayanan GPU zuwa cikakken kwafin da duk GPUs ke riƙe?
Duk-taro na tattara keɓaɓɓun guda daga kowane GPU kuma suna tattara cikakken saiti akan su duka, ana amfani da su sosai a cikin horon sharar fage kamar FSDP.
Menene NVIDIA SHARP ke yi don ayyukan gama kai?
SHARP yana yin lissafin cikin hanyar sadarwa, yana yin wani ɓangare na raguwa a cikin maɓalli don haka dole ne ƙasan bayanai su ketare hanyoyin haɗin gwiwa, yana haɓaka ƙungiyoyin haɗin gwiwa.