Jokkoo bu mbooloo ak NCCL
Jokkoo bu mbooloo mooy anam wi kuréelu GPU di weccoo ak boole ay done, te NCCL mooy bibliotek bu NVIDIA biy tax weccoo yooyu gaaw lool.
Résumé
Operations like all-reduce are the heartbeat of distributed training, synchronizing gradients across every GPU each step.
Plongeur bu xóot
Taggat ab model bu mag dafay tekki ni GPU bu nekk dafay xayma ay gradient ci ay done boppam, kon GPU yépp dañu wara déggoo ci benn resultaa buñ boole balaa ñuy jël jéego bu ci topp. Koordinaasioŋ boobu ñu ngi koy defee ci liggéey yu mbooloo: wàññi lépp ci valeur yi ci GPU yi, ba noppi jox ku nekk resultaa bi; all-gather dafay dajale piyeesu GPU bu nekk ci benn kopi bu mat sëkk ci ñoom ñépp; broadcast dafay yónnee benn GPU ci yeneen yi; wàññi-tasaaroo boole daal di xaaj. NCCL (Bibliothèque de communication collective de NVIDIA) dafay jëfandikoo lii ci anam wu jaar yoon ci GPU yi ci serwër bi ak ci serwër yi, di jëfandikoo algorithm yu xam topologie yu melni ring ak garab all-reduce. Dafay jëfandikoo NVLink ci biir benn node ak InfiniBand wala RoCE diggante node yi, te mooy yax giy jokkoo ci suufu PyTorch DDP, FSDP, DeepSpeed, ak Megatron.
Gis-gis xarala
Ring all-reduce mooy algorithm bu yàgg bi: GPU yi dañuy forme benn ring logique, ba noppi done yi dañu leen xaaj ay pàcc yuy wër suko defee jéego bu nekk di jaxasoo ci jokkoo, loolu mooy tax bandwidth bi gëna baax te amul benn njeexte ci limu GPU yi. Ci node yu bari, algorithm yu sukkandiko ci garab dañuy wàññi latency ci boole resultaa yi ci hierarchie. NCCL dafay gis topologie bi ci boppam, tànn algorithm bi gëna baax, te mën na wàññi math bi ci reso bi ak NVIDIA SHARP, di xaaj done yi wara jaar ci lëkkalekaay yi.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu jokkoo bu mbooloo ak NCCL
Ginaaw clusters yi dañuy gëna yokk ba yegg ci téemeeri junni GPU, jokkoo dafay gëna ëpp doole ci diiru tàggat yaram, kon bibliotek yu mbooloo yi dañuy nekk frontiere bu tàng. Xaarandil ordinatër bu gëna xóot ci biir reso bi (switch yi ñooy wàññi), ordinatër ak jokkoo bu gëna baax ngir nëbb latency, ak collectives yu gëna néew precision yuy wàññi bytes yiñ toxal. Competition bi dafay yokk itam, ak jeegoowu jaaykat yu bari ak RDMA bu Ethernet di push yeneen pexe, ci noonu la NCCL di wéy di gëna dëgëral lëkkaloo ak NVLink, NVSwitch, ak tissu optique yuy génn.
Doxal ci àdduna dëgg
Dañuy boole gradient yi ci bépp jéego bu ñuy tàggat ci GPU yépp di jëfandikoo wàññi lépp ci PyTorch DistributedDataParallel
Xaajale gëna xéewale réew yi ak dajale ay parametre ci laaj ak dajale lépp ak wàññi tasaaroo ci FSDP wala DeepSpeed ZeRO
Diffuser diisaayu model bi njëkk ci benn GPU dem ci ñeneen ñi yépp ci ndoortelu tàggat yaram
Jëfandikoo ring ngir wàññi lépp ci NVLink ak InfiniBand ngir tëye yaatuwaayu band bi ci biir cluster GPU yu bari
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Collective Communication and NCCL quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Man-man yi ci net bi ak yi nekkul ci net bi
Laaj yi ñuy faral di laaj
What is Collective Communication and NCCL?
Jokkoo bu mbooloo mooy anam wi kuréelu GPU di weccoo ak boole ay done, te NCCL mooy bibliotek bu NVIDIA biy tax weccoo yooyu gaaw lool. Liggéeyukaay yu melni all-reduce mooy beat xol bu tàggat buñ séddale, di boole gradient yi ci bepp GPU jéego bu nekk.
Lan la operaasioŋ buy wàññi lépp di def ci tàggat-yaram buñ séddale?
All-reduce dafay boole (wala boole) benn valeur ci GPU bu nekk ba noppi séddale resultaa bi nuróo ci ñoom ñépp, te loolu mooy anam wi ñuy synchronise gradient yi.
Luy tekki akronim NCCL?
NCCL mooy Bibliothèque de communication collective bu NVIDIA, mooy jëfandikoo ay GPU yu bari ak ay node yu bari yu gaaw.
Lan moo waral ñu jàpp ni ring bi dafay wàññi bandwidth bi gëna baax?
Soo xaajalee done yi ak joxe ay piyees ci benn ring logique, lëkkalekaay bu nekk dañu koy jëfandikoo benn yoon, ba noppi toxal gi dafay nekk lu amul benn njeexte ci limu GPU yi.
Ban liggéey mooy dajale done yu GPU bu nekk ci benn kopi bu mat bu GPU yépp yor?
All-gather dafay dajale piyees yu wuute yi ci GPU bu nekk ba noppi dajale ensemble bi yépp ci ñoom ñépp, ñu koy jëfandikoo bu baax ci tàggat yaram bu melni FSDP.
Lan la NVIDIA SHARP di def ci liggéey yu mbooloo?
SHARP dafay def ordinatër ci biir reso bi, di def wàll ci wàññi gi ci biir commutateur bi, suko defee done yu néew yi wara jaar ci lëkkalekaay yi, di gaawlu collectives yi.