Ndekota data
Myirịta data na-azụ otu ụdị ngwa ngwa site n'ịmegharị ya n'ofe ọtụtụ GPU, na-eji GPU nke ọ bụla na-ahazi ibe dị iche iche nke batch data.
Nchịkọta
It is the workhorse technique that lets teams scale to dozens or thousands of accelerators.
Ime miri emi
Na myirịta data, GPU ọ bụla na-ejide otu ihe atụ nke ihe atụ mana ọ na-ahazi obere obere ihe atụ ọzụzụ. Ngwaọrụ ọ bụla na-agbakọ ngafe na-aga n'ihu na azụ azụ n'onwe ya, na-emepụta usoro gradients nke ya. Tupu mmelite arọ, a na-agbakọ gradients n'ofe GPU niile site na iji ọrụ nkwurịta okwu na-ebelata ihe niile, yabụ oyiri ọ bụla na-anọ na mmekọrịta ma na-eme ka a ga-asị na ọ zụrụ ya n'otu nnukwu ngwakọta. Nke a na-abawanye mmepụta nke ọma: 8 GPU nwere ike ịta site na 8x data kwa nzọụkwụ. Ihe ejidere bụ na GPU nke ọ bụla ga-adabarịrị n'ụdị niile, gradients ya, na ọnọdụ njikarịcha na ebe nchekwa, yabụ ndakọrịta data doro anya anaghị enyere aka mgbe ihe nlereanya buru oke ibu maka otu ngwaọrụ.
Nghọta nka nka
Isi ọrụ a na-ebelata ihe niile, nke na-achịkọta gradients n'ofe ngwaọrụ wee kesaa nsonaazụ ya. Mgbanaka mbelata niile, nke ụlọ akwụkwọ dị ka NCCL na Horovod na-eji, na-agafe obere gradient gburugburu mgbanaka ezi uche dị na ya, nkwurịta okwu zuru ezu na-adabere na ọnụ ọgụgụ GPU. PyTorch's DistributedDataParallel na-ekpuchi nzikọrịta ozi a na ngafe azụ azụ, na-agbanyụ mmekọrịta gradient maka ọkwa mbụ ka ọkwa ndị ọzọ ka na-agbakọ, na-ezochi ọtụtụ nkwụsị netwọkụ.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke Data Parallelism
A na-ejikọta myirịta data dị ọcha yana nchacha na myirịta ụdị n'ime atụmatụ 'nD parallelism' ngwakọ maka ụdị trillion-parameter. Na-atụ anya mkpakọ gradient dị mma karịa, nkwukọrịta asynchronous na kpuchiri ekpuchi, yana topology maara ihe niile na-ebelata nke na-erigbu NVLink ngwa ngwa n'ime ọnụ yana jiri nwayọ InfiniBand gafee ọnụ. Ka ụyọkọ na-eto, ibelata nzikọrịta ozi-na-mgbakọ bụ ihe ịma aka nke etiti maka ime ka ọtụtụ puku GPU na-arụ ọrụ.
Mmejuputa n'ezie n'ụwa
Ọzụzụ nhazi ọkwa ihe onyonyo ResNet gafee 8 GPU n'otu ihe nkesa na-eji PyTorch DistributedDataParallel, GPU ọ bụla na-ejikwa 32 nke ihe onyonyo 256.
Ịmalite ọzụzụ BERT n'ofe narị GPU na Horovod, na-eji mgbanaka mbelata iji mekọrịta gradients nzọụkwụ ọ bụla.
Nhazi nke ọma ụdị nkwanye na ụyọkọ ọnụ ọnụ ọtụtụ ebe ọnụ ọnụ nke ọ bụla na-arụ ọrụ dị iche iche na-emekọrịta ihe.
Iji TensorFlow's MirroredStrategy gbasaa ọzụzụ nke ụdị ọhụụ n'ofe ọtụtụ GPU n'otu ebe ọrụ nwere obere mgbanwe koodu.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Data Parallelism quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Njikwa data AI
Ajụjụ a na-ajụkarị
What is Data Parallelism?
Myirịta data na-azụ otu ụdị ngwa ngwa site n'ịmegharị ya n'ofe ọtụtụ GPU, na-eji GPU nke ọ bụla na-ahazi ibe dị iche iche nke batch data. Ọ bụ usoro ịnya ọrụ na-eme ka ndị otu na-agba ọsọ ruo ọtụtụ iri ma ọ bụ puku kwuru puku ngwa ngwa.
N'ụdị data ọkọlọtọ, kedu ihe GPU nke ọ bụla na-ejide?
GPU nke ọ bụla na-edobe ihe nlere zuru oke ma na-ahazi akụkụ dị iche iche nke batch data, nke bụ ihe na-eme ka ọ bụrụ 'data' myirịta kama ịmekọrịta ihe nlereanya.
Kedu ọrụ nzikọrịta ozi na-eme ka ihe nlere anya mekọrịta usoro ọ bụla?
Mgbe ngafe azụ ọ bụla gasịrị, a na-ejikọta gradients n'ofe ngwaọrụ site na mbelata niile (nke a na-achịkọta ya na nkezi) yabụ oyiri ọ bụla na-emetụta otu mmelite ahụ.
Kedu ihe bụ isi njedebe nke myirịta data doro anya?
N'ihi na GPU ọ bụla na-ejide ihe niile zuru oke, myirịta data adịghị eme ihe ọ bụla iji nyere aka mgbe ihe nlereanya dị nnọọ ukwuu nke dabara na otu ngwaọrụ.
Kedu ihe kpatara mgbanaka ji ebelata ihe niile mara mma maka ọnụ ọgụgụ GPU buru ibu?
Kpọọ ihe na-ebelata ihe niile na-agafe gradient chunks gburugburu mgbanaka ezi uche dị na ya, ya mere mkpokọta bandwit GPU ọ bụla na-eziga na-anọgide na-adịgide adịgide n'agbanyeghị ole GPU na-esonye.
Kedu ka PyTorch DistributedDataParallel si ezochi oge nzikọrịta ozi?
DDP na-amalite ịmekọrịta gradients maka ọkwa mbụ ebe a ka na-agbakọ ọkwa ndị ọzọ, na-ejikọta nzikọrịta ozi netwọk na mgbakọ.