Kuringaniza amakuru
Data parallelism itoza icyitegererezo cyihuse mukwigana muri GPU nyinshi, hamwe na GPU itunganya ibice bitandukanye byamakuru.
Incamake
It is the workhorse technique that lets teams scale to dozens or thousands of accelerators.
Kwibira cyane
Muburinganire bwamakuru, buri GPU ifite kopi imwe yuburemere bwikitegererezo ariko itunganya mini-batike yingero zamahugurwa. Buri gikoresho kibara imbere ninyuma yinyuma yigenga, itanga umurongo wa gradients. Mbere yo kuvugurura ibipimo, gradients zigereranijwe muri GPU zose ukoresheje ibikorwa byose bigabanya itumanaho, bityo buri kopi iguma muri sync kandi ikitwara nkaho yatojwe kumurongo umwe uhuriweho. Ibi bigwiza neza ibyinjira: 8 GPUs irashobora guhekenya hafi 8x amakuru kuri buri ntambwe. Gufata ni uko buri GPU igomba guhuza icyitegererezo cyose, gradients zayo, hamwe na optimizer leta murwibutso, kubwibyo rero amakuru asobanutse ntagufasha mugihe icyitegererezo ari kinini kubikoresho bimwe.
Ubushishozi
Igikorwa cyingenzi ni byose-bigabanya, bikusanya gradients hejuru yibikoresho kandi bigabura ibisubizo. Impeta yose-igabanya, ikoreshwa namasomero nka NCCL na Horovod, itambutsa uduce duto duto tuzengurutse impeta yumvikana kuburyo itumanaho ryose ritigenga kubara GPU. PyTorch's DistributedDataParallel ihuza itumanaho hamwe ninyuma yinyuma, ikuraho sync ya sync ya gradient kubice byambere mugihe nyuma ibice biracyabazwe, bihisha byinshi mubukererwe bwurusobe.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza h'amakuru abangikanye
Kubangikanya amakuru yuzuye bigenda bihuzwa no gutondeka no kugereranya icyitegererezo muburyo bwa Hybrid 'nD parallelism' kubikorwa bya tiriyari-ibipimo. Tegereza ubwenge bworoheje buhoro buhoro, itumanaho ridahuje kandi ryuzuye, hamwe na topologiya-byose-bigabanya-bigabanya gukoresha NVLink yihuta muri node kandi gahoro gahoro InfiniBand kuri node. Mugihe amahuriro akura, kugabanya itumanaho-kuri-kubara bikomeje kuba ingorabahizi yubuhanga bwo gukomeza ibihumbi bya GPU.
Gushyira mu bikorwa Isi
Guhugura ResNet ishusho itondekanya muri GPUs 8 muri seriveri imwe ukoresheje PyTorch IkwirakwizwaDataParallel, buri GPU ikora 32 muri 256-ishusho.
Gupima BERT kwitegereza amagana ya GPU hamwe na Horovod, ukoresheje impeta byose-kugabanya kugirango uhuze gradients buri ntambwe.
Kuringaniza neza icyitegererezo cyicyitegererezo kuri cluster-node aho buri node itunganya abakoresha-imikoranire itandukanye.
Koresha MirroredStrategy ya TensorFlow kugirango ukwirakwize amahugurwa yicyitegererezo muri GPU nyinshi kumurongo umwe wakazi hamwe nimpinduka ntoya.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Data Parallelism quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Imiyoborere ya AI
Ibibazo bikunze kubazwa
What is Data Parallelism?
Data parallelism itoza icyitegererezo cyihuse mukwigana muri GPU nyinshi, hamwe na GPU itunganya ibice bitandukanye byamakuru. Nubuhanga bwakazi bukora butuma amakipe apima kugera kubihumbi cyangwa ibihumbi byihuta.
Mubisanzwe amakuru abangikanye, buri GPU ifata iki?
Buri GPU ibika kopi yuzuye yicyitegererezo kandi igatunganya igice cyihariye cyamakuru yatanzwe, aricyo gituma 'data' ibangikanya aho kuba icyitegererezo.
Ni ubuhe buryo bwo gutumanaho butuma icyitegererezo gikoporora buri ntambwe?
Nyuma ya buri cyuma gisubira inyuma, gradients zahujwe kubikoresho binyuze muri byose bigabanya (mubisanzwe byavuzwe haruguru noneho ugereranije) kuburyo buri kopi ikoresha ivugurura rimwe.
Ni ubuhe butumwa nyamukuru bwo guhuza amakuru asanzwe?
Kuberako buri GPU ifite kopi yuzuye yibintu byose, parallelism ntacyo ikora kugirango ifashe mugihe icyitegererezo ari kinini cyane kuburyo kidahuza igikoresho kimwe.
Ni ukubera iki impeta zose zigabanya gukurura umubare munini wa GPU?
Impeta byose-bigabanya passes gradient uduce twizengurutse impeta yumvikana, bityo umurongo mugari wa buri GPU wohereje uhoraho utitaye kumubare GPU yitabira.
Nigute PyTorch IkwirakwizwaDataParllel ihisha itinda ryitumanaho?
DDP itangira guhuza gradients kubice byambere mugihe nyuma ibice biracyabarwa, guhuza imiyoboro y'itumanaho hamwe no kubara.