Paralelismu done
Parallelism ci done yi dafay gëna gaaw ci tàggat benn model ci ni ñu koy toppandoo ci GPU yu bari, GPU bu nekk di liggéey ci wàll wu wuute ci done yi.
Résumé
Mooy pexem workhorse biy may ekip yi ñu mëna yegg ba fukki-fukki wala junni-junni accelerator.
Plongeur bu xóot
Ci parallelism done, GPU bu nekk dafay yor benn kopi bu nuróo ci diisaayu model bi waaye dafay liggéey ci misaali tàggat yu ndaw yu wuute. Aparey bu nekk dafay xayma paas bi ci kanam ak ci ginaaw boppam, ba noppi defar ay gradient boppam. Laata ñuy yeesal poid yi, gradient yi dañu leen di moyenne ci GPU yépp di jëfandikoo ab jokkoo buy wàññi lépp, kon replica bu nekk dafay des ci sync ba noppi di doxalee ni dafa tàggat ci benn batch bu mag buñ boole. Loolu dafay yokk produit yi: 8 GPUs mën nañu sàqami lu tollu ci 8x done ci jéego bu nekk. Japp bi mooy GPU bu nekk dafa wara méngoo ak model bi yépp, ay gradient, ak nekkinu optimisatër bi ci mémoire bi, kon parallelism done bu leer du jàppale sudee model bi dafa rëy lool ci benn aparey.
Gis-gis xarala
Li gëna am solo mooy wàññi lépp, muy boole gradient yi ci aparey yi ba noppi séddalewaat li ci génn. Ring all-reduce, bibliotek yu melni NCCL ak Horovod di jëfandikoo, dafay romb ay gradient yu wër benn ring bu logic suko defee jokkoo bu mat sëkk nekkul ci lim GPU. DistributedDataParallel bu PyTorch dafay jaxasoo jokkoo bi ak paas bi ci ginaaw, di génne sync gradient ci diisaay yu njëkk yi fekk diisaay yu ci topp yi ñu ngi nekk ci ordinatër, di nëbb lu bari ci latency reso bi.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu paralelismu done
Paralelismu done yu sell dañuy gëna boole ak xaaj ak paralelismu model ci pexe hybrid 'nD paralelism' ngir model yu am ay bilioŋu parametre. Xaarandil kompresioŋ gradient bu gëna am xel, jokkoo bu asynchrone ak buy jaxasoo, ak topologie-xam lépp-waññi buy jëfandikoo NVLink bu gaaw ci biir benn node ak InfiniBand bu yeex ci biir node yi. Lu clusters yi di màgg, wàññi ratio bi diggante jokkoo ak ordinatër mingi wéy di nekk jafe-jafe bi gëna mag ci ingenieur yi ngir tëye ay junni GPU yu liggéey.
Doxal ci àdduna dëgg
Taggat benn ResNet buy xaaj nataal ci 8 GPU ci benn serwër buy jëfandikoo PyTorch DistributedDataParallel, GPU bu nekk di jëfandikoo 32 ci 256 nataal.
Eskalaasioŋ BERT bi ñuy njëkka tàggat ci téemeeri GPU ak Horovod, jëfandikoo ring all-reduce ngir méngale gradient yi jéego bu nekk.
Defar ab xeetu xalaat ci cluster bu bari node fu node bu nekk di doxal ay shards yu wuute ci diggante jëfandikukat yi.
Jëfandikoo TensorFlow's MirroredStrategy ngir tasaare tàggat ci xeetu gis-gis ci GPU yu bari ci benn station de travail ak coppite kode yu néew.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Data Parallelism quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Nguuru Done IA
Laaj yi ñuy faral di laaj
Luy paralelismu done?
Parallelism ci done yi dafay gëna gaaw ci tàggat benn model ci ni ñu koy toppandoo ci GPU yu bari, GPU bu nekk di liggéey ci wàll wu wuute ci done yi. Mooy pexem workhorse biy may ekip yi ñu mëna yegg ba fukki-fukki wala junni-junni accelerator.
Ci parallelism done buñ miin, lan la GPU bu nekk di tëye?
GPU bu nekk dafay denc koppi bu mat sëkk ci model bi ba noppi di liggéey ci wàll wu wuute ci done yi, moo tax mu nekk parallelism 'done' moo gën parallelism model.
Ban jëfu jokkoo mooy tëye replica yu model yi ci jéego bu nekk?
Ginaaw bépp dellu ginaaw, gradient yi dañuy boole ci aparey yi jaaraleko ci all-reduce (dañu koy boole ba noppi moyenne) suko defee replica bu nekk di jëfandikoo benn yeesal bi.
Lan mooy gàttal bu mag bi ci parallelism done yu leer?
Ndax GPU bu nekk dafay tëye kopi bu mat sëkk ci lépp, parallelism done du def dara luy jàppale sudee model bi dafa yaatu lool ba mënul nekk ci benn aparey.
Lan moo waral ring all-reduce di xëcc GPU yu bari?
Ring all-reduce dafay romb ay gradient yu wër benn ring logique, kon bandwidth bi GPU bu nekk di yónnee du soppeeku, ak limu GPU yi ci bokk.
naka la ko PyTorch di nëbbee ay done yuñ séddale parallel?
DDP mingi tàmbali synchroniser gradient yi ngir layers yu njëkk ya, fekk layers yu ci topp yi ñu ngi doon calculer, di jaxasoo jokkoo reso bi ak calcul bi.