Data Parallelism
Ibaṣepọ data ṣe ikẹkọ awoṣe kan ni iyara nipa ṣiṣatunṣe rẹ kọja ọpọlọpọ awọn GPUs, pẹlu ṣiṣe GPU kọọkan ni bibẹ pẹlẹbẹ ti o yatọ ti ipele data naa.
Akopọ
It is the workhorse technique that lets teams scale to dozens or thousands of accelerators.
Jin Dive
Ni afiwe data, gbogbo GPU ni ẹda kanna ti awọn iwuwo awoṣe ṣugbọn ṣe ilana iwọn-kekere pato ti awọn apẹẹrẹ ikẹkọ. Ẹrọ kọọkan ṣe iṣiro iwe-iwọle siwaju ati sẹhin ni ominira, ti n ṣe agbejade eto awọn gradients tirẹ. Ṣaaju imudojuiwọn awọn iwuwo, awọn iwọn gradients jẹ aropin kọja gbogbo awọn GPUs ni lilo iṣẹ ibaraẹnisọrọ gbogbo-dinku, nitorinaa gbogbo ajọra duro ni amuṣiṣẹpọ ati huwa bi ẹnipe o ṣe ikẹkọ lori ipele idapọpọ nla kan. Eyi ṣe isodipupo imunadoko: Awọn GPU 8 le jẹ nipasẹ aijọju 8x data fun igbesẹ kan. Apeja ni pe GPU kọọkan gbọdọ baamu gbogbo awoṣe, awọn gradients rẹ, ati ipo iṣapeye ni iranti, nitorinaa afiwera data itele ko ṣe iranlọwọ nigbati awoṣe ba tobi ju fun ẹrọ ẹyọkan.
Imọ-imọ-ẹrọ
Iṣiṣẹ bọtini jẹ gbogbo-dinku, eyiti o ṣe akopọ awọn gradients kọja awọn ẹrọ ati pinpin abajade. Ohun orin gbogbo-din, ti a lo nipasẹ awọn ile-ikawe bii NCCL ati Horovod, kọja awọn ege gradient ni ayika iwọn ọgbọn kan nitorina ibaraẹnisọrọ lapapọ jẹ ominira ti kika GPU. PyTorch's DistributedDataParallel ṣakojọpọ ibaraẹnisọrọ yii pẹlu iwọle sẹhin, fifipa mimuuṣiṣẹpọ gradient kuro fun awọn ipele ibẹrẹ lakoko ti awọn fẹlẹfẹlẹ nigbamii tun n ṣe iṣiro, fifipamo pupọ ti aipe nẹtiwọọki.
Ipa Ilana
Iye owo ati isuna
Awọn ipinnu faaji ṣe awakọ iṣẹ ati idiyele iṣẹ fun awọn ọdun.
Awọn ipinnu diẹ sii
Ẹkọ imọ-ẹrọ ṣe iranlọwọ fun awọn ẹgbẹ lati yan akopọ to tọ, kii ṣe ọkan tuntun nikan.
Iṣakoso didara
Awọn yiyan imọ-ẹrọ to dara julọ dinku awọn iṣẹlẹ igbẹkẹle ni iṣelọpọ.
Ojo iwaju ti Data Parallelism
Ibaṣepọ data mimọ ti n pọ si pọ si pẹlu sharding ati afiwera awoṣe sinu awọn ilana ‘parallelism nD’ arabara fun awọn awoṣe paramita aimọye. Reti funmorawon gradient ijafafa, asynchronous ati ibaraenisọrọ agbekọja, ati topology-mọ gbogbo-dinku ti o lo NVLink yiyara laarin apa kan ati ki o lọra InfiniBand kọja awọn apa. Bi awọn iṣupọ ṣe ndagba, idinku ipin ibaraẹnisọrọ-si-iṣiro jẹ ipenija imọ-ẹrọ aarin fun mimu ẹgbẹẹgbẹrun awọn GPU ṣiṣẹ lọwọ.
Real-World imuse
Ikẹkọ olupilẹṣẹ aworan ResNet kọja awọn 8 GPUs ninu olupin kan nipa lilo PyTorch DistributedDataParallel, GPU kọọkan mimu 32 ti ipele aworan 256 kan.
Ilọsiwaju ikẹkọ BERT kọja awọn ọgọọgọrun ti GPUs pẹlu Horovod, ni lilo ohun gbogbo-dinku lati muu awọn gradients ṣiṣẹpọ ni igbesẹ kọọkan.
Titun-tunse awoṣe iṣeduro lori iṣupọ-opopona pupọ nibiti ipade kọọkan ti n ṣe ilana oriṣiriṣi awọn shards ibaraenisepo olumulo.
Lilo TensorFlow's MirroredStrategy lati tan ikẹkọ ti awoṣe iran kọja ọpọ GPUs lori ibi iṣẹ kan pẹlu awọn ayipada koodu pọọku.
Awọn ewu & Awọn ọna iṣọ
Ṣiṣepe ala-ilẹ kan le tọju awọn ailagbara eto ti o gbooro.
Awọn ohun elo amayederun ati awọn idiyele itọju nigbagbogbo ni aibikita.
Aabo ati awọn ela akiyesi le dagba bi awọn eto ṣe di eka sii.
Ilana Ilana imuse
Ṣetumo lairi, didara, ati awọn ibi-afẹde idiyele ṣaaju imuse.
Aṣepari labẹ ẹru ojulowo ati awọn ipo data.
Abojuto ohun elo fun awọn aṣiṣe, fiseete, ati ipa olumulo.
Mura ipadasẹhin pada ati awọn ipa ọna esi iṣẹlẹ ṣaaju iwọn.
Tesiwaju Ṣiṣawari
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Data Parallelism quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Itọsọna atẹle
AI Data Isakoso
Awọn ibeere ti a beere nigbagbogbo
What is Data Parallelism?
Ibaṣepọ data ṣe ikẹkọ awoṣe kan ni iyara nipa ṣiṣatunṣe rẹ kọja ọpọlọpọ awọn GPUs, pẹlu ṣiṣe GPU kọọkan ni bibẹ pẹlẹbẹ ti o yatọ ti ipele data naa. O jẹ ilana iṣẹ-iṣẹ ti o jẹ ki awọn ẹgbẹ ṣe iwọn si awọn dosinni tabi ẹgbẹẹgbẹrun awọn iyara iyara.
Ni afiwe data boṣewa, kini GPU kọọkan mu?
GPU kọọkan n tọju ẹda ni kikun ti awoṣe ati awọn ilana ipin kan pato ti ipele data, eyiti o jẹ ki o jẹ 'data' afiwera kuku ju afiwe awoṣe lọ.
Iṣiṣẹ ibaraẹnisọrọ wo ni o tọju awọn ẹda awoṣe ni mimuṣiṣẹpọ ni igbesẹ kọọkan?
Lẹhin igbasilẹ ẹhin kọọkan, awọn gradients ti wa ni idapo kọja awọn ẹrọ nipasẹ gbogbo-dinku (eyiti o ṣe akopọ lẹhinna aropin) nitorina gbogbo ẹda kan lo imudojuiwọn kanna.
Kini aropin akọkọ ti itele data parallelism?
Nitoripe gbogbo GPU ni ẹda kikun ti ohun gbogbo, parallelism data ko ṣe nkankan lati ṣe iranlọwọ nigbati awoṣe kan ba tobi ju lati baamu lori ẹrọ kan.
Kini idi ti oruka gbogbo-din ṣe wuni fun awọn iṣiro GPU nla?
Ohun orin gbogbo-din gba koja gradient chunks ni ayika kan mogbonwa oruka, ki lapapọ bandiwidi kọọkan GPU rán duro ibakan laibikita bawo ọpọlọpọ awọn GPUs kopa.
Bawo ni PyTorch DistributedDataParallel ṣe tọju lairi ibaraẹnisọrọ bi?
DDP bẹrẹ mimuuṣiṣẹpọ awọn gradients fun awọn ipele iṣaaju lakoko ti awọn fẹlẹfẹlẹ nigbamii ti wa ni iṣiro, ibaraẹnisọrọ nẹtiwọọki agbekọja pẹlu iṣiro.