Daidaiton Bayanai
Daidaituwar bayanai yana horar da ƙira ɗaya cikin sauri ta hanyar yin kwafinsa a cikin GPUs da yawa, tare da kowane GPU yana sarrafa wani yanki na daban na tsarin bayanai.
Dubawa
It is the workhorse technique that lets teams scale to dozens or thousands of accelerators.
Zurfafa nutsewa
A cikin daidaiton bayanai, kowane GPU yana riƙe da kwafi iri ɗaya na ma'aunin ƙirar amma yana aiwatar da takamaiman ƙaramin misalan horo. Kowace na'ura tana ƙididdige wucewar gaba da baya da kanta, tana samar da nata tsarin na gradients. Kafin sabuntawar ma'auni, ana ƙididdige matakan gradients a duk GPUs ta amfani da aikin sadarwa mai rahusa, don haka kowane kwafi yana kasancewa cikin aiki tare kuma yana nuna kamar an horar da shi akan babban haɗin gwiwa. Wannan yana haɓaka kayan aiki yadda ya kamata: 8 GPUs na iya taunawa ta hanyar kusan 8x bayanan kowane mataki. Abin kamawa shine kowane GPU dole ne ya dace da duka ƙirar, gradients, da yanayin ingantawa a cikin ƙwaƙwalwar ajiya, don haka daidaitattun bayanai ba ya taimakawa lokacin da samfurin ya yi girma ga na'ura ɗaya.
Fahimtar Fasaha
Makullin aiki shine duka-raguwa, wanda ke tattara gradients a cikin na'urori kuma yana sake rarraba sakamakon. Rage zobe, da dakunan karatu kamar NCCL da Horovod ke amfani da su, suna wucewa da ƙugiya a kusa da zobe mai ma'ana don haka jimillar sadarwa ta kasance mai zaman kanta daga ƙididdigar GPU. PyTorch's DistributedDataParallel ya mamaye wannan hanyar sadarwa tare da wucewar baya, yana kashe daidaitawar gradient don farkon yadudduka yayin da yadudduka ke ci gaba da yin lissafi, suna ɓoye yawancin lattin hanyar sadarwa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Daidaiton Bayanai
Daidaitaccen daidaiton bayanai yana ƙara haɗawa tare da rarrabuwar kawuna da ƙima a cikin dabarun 'nD parallelism' gauraye don ƙirar siga tiriliyan. Yi tsammanin matsi mafi wayo, asynchronous da madaidaicin sadarwa, da topology-sane da duk-rage wanda ke cin gajiyar NVLink cikin sauri a cikin kumburi da sannu a hankali InfiniBand a kan nodes. Yayin da gungu ke girma, rage ƙimar sadarwa-zuwa-ƙididdigar ƙididdiga ya kasance babban ƙalubalen injiniya don kiyaye dubban GPUs cikin aiki.
Aiwatar da Gaskiyar Duniya
Horar da mai rarraba hoto na ResNet a cikin 8 GPUs a cikin sabar guda ɗaya ta amfani da PyTorch DistributedDataParallel, kowane GPU yana sarrafa 32 na tsari mai hoto 256.
Ƙimar BERT pretraining a cikin ɗaruruwan GPUs tare da Horovod, ta amfani da duk-rage don aiki tare gradients kowane mataki.
Kyakkyawan daidaita samfurin shawarwari akan gungu mai nau'in kumburi inda kowane kumburi yana aiwatar da shards-mu'amala daban-daban.
Amfani da Dabarun Mirrored na TensorFlow don yada horon ƙirar hangen nesa a cikin GPUs da yawa akan wurin aiki guda ɗaya tare da ƙaramin canje-canje na lamba.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Data Parallelism quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
AI Data Gudanarwa
Tambayoyin da ake yawan yi
What is Data Parallelism?
Daidaituwar bayanai yana horar da ƙira ɗaya cikin sauri ta hanyar yin kwafinsa a cikin GPUs da yawa, tare da kowane GPU yana sarrafa wani yanki na daban na tsarin bayanai. Dabarar dokin aiki ce ke baiwa ƙungiyoyin damar yin ƙima zuwa dubunnan ko dubunnan masu hanzari.
A daidaitaccen daidaiton bayanai, menene kowane GPU ke riƙe?
Kowane GPU yana adana cikakken kwafi na samfurin kuma yana aiwatar da wani sashe na musamman na tsarin bayanai, wanda shine abin da ya sa ya zama daidaici na 'bayanai' maimakon daidaiton samfuri.
Wane aikin sadarwa ne ke kiyaye kwafin samfurin a daidaita kowane mataki?
Bayan kowace wucewa ta baya, ana haɗa gradients a cikin na'urori ta hanyar duk-rage (yawanci taƙaitawa sannan matsakaita) don haka kowane kwafi yana aiki da sabuntawa iri ɗaya.
Menene babban iyakancewar daidaitattun bayanai?
Saboda kowane GPU yana riƙe da cikakken kwafin komai, daidaiton bayanai ba ya yin komai don taimakawa lokacin da ƙirar ta yi girma sosai don dacewa da na'ura ɗaya.
Me yasa duk-raguwar zobe ke da kyau ga manyan ƙididdigar GPU?
Ringing duk-rage yana wucewa ƙuƙumi a kusa da zobe mai ma'ana, don haka jimillar bandwidth kowane GPU ya aika yana tsayawa ko da kuwa nawa GPUs ke shiga.
Ta yaya PyTorch DistributedDataParallel ke ɓoye jinkirin sadarwa?
DDP ta fara aiki tare da gradients don matakan farko yayin da har yanzu ana ƙididdige yadudduka, sadarwar hanyar sadarwa tare da lissafi.