Ukufana kweModel kanye nePipeline
Uma imodeli inkulu kakhulu ukuthi ingangena ku-GPU eyodwa, imodeli nokufana kwepayipi kuhlukanisa imodeli ngokwayo kuwo wonke amadivayisi.
Uhlolojikelele
This is what makes training giant language models with hundreds of billions of parameters physically possible.
I-Deep Dive
Ukufana kwemodeli kuhlukanisa imodeli eyodwa kuwo wonke ama-GPU amaningi ukuze kungabikho idivayisi eyodwa edinga ukubamba zonke izisindo. Kunezinhlobo ezimbili zokunambitheka eziyinhloko. I-Tensor (intra-layer) parallelism ihlukanisa izibalo ngaphakathi kwesendlalelo, njengokusika ukuphindaphinda okukhulu kwe-matrix kuwo wonke ama-GPU okuthi ingxenye ngayinye ibale ingxenye yalokho okukhiphayo. Ukufana kwepayipi (inter-layer) kwabela izendlalelo ezihlukene ezilandelanayo kuma-GPU ahlukene, ngakho-ke ibhulokhi engu-1 iphila ku-GPU 0, i-block 2 ku-GPU 1, njalo njalo, ngokuvula okudlulele phambili njengomugqa wokuhlanganisa. Inselele ngamapayipi okungenangqondo 'ibhamuza': kuyilapho i-GPU 0 isebenza kuqeqebana lokuqala, ama-GPU angezansi ahlala engenzi lutho. Ukufakwa kwamapayipi kuhlukanisa iqoqo ngalinye libe ngamaqoqo amancane ukuze zonke izigaba zihlale zimatasa, kuthuthukisa kakhulu ukusetshenziswa.
I-Technical Insight
I-Tensor parallelism (njengaku-NVIDIA Megatron-LM) ihlukanisa ikholomu ka-matrices wesisindo- noma ngokuhlakanipha komugqa futhi isebenzisa ukunciphisa konke ukuze kuhlanganiswe imiphumela engaphelele, igcine ukuxhumana ngaphakathi kwenodi ye-NVLink esheshayo. Ukufana kwepayipi (i-GPipe, i-PipeDream) ihlukanisa inqwaba ibe amaqoqo amancane ageleza ngezigaba ngeshejuli ehlukanisiwe, incipha isikhathi 'sebhamuza' sokungenzi lutho. Okubili kuvame ukugqitshwa ndawonye, nokufana kwe-tensor phakathi kwe-node nokufana kwepayipi kuwo wonke ama-node.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Lemodeli Nokufana Kwepayipi
Izinhlaka ziya ngokuya zenza kube ngokuzenzakalela inkinga enzima yokunquma indlela yokuhlukanisa imodeli kuwo wonke amadivayisi, kusetshenziswa ukwenza iphrofayela nokusesha ukulinganisa ikhompuyutha nokuxhumana. Lindela ukuhlanganiswa okuqinile kwe-tensor, ipayipi, nokufana kwedatha (i-3D parallelism), ukuhlela i-micro-batch ehlakaniphile ukuze kucishe kuqede amabhamuza epayipi, kanye nezingxenyekazi zekhompuyutha ezinokuxhuma okusheshayo ukuze ukuhlukanisa isendlalelo esisodwa kuwo wonke ama-chip kube kushibhe futhi kube umkhuba kakhudlwana kumamodeli amakhudlwana njalo.
Ukuqaliswa Komhlaba Wangempela
Ukuqeqesha amamodeli esitayela se-GPT nge-NVIDIA Megatron-LM, ehlukanisa ukunaka kwesendlalelo ngasinye se-transformer namatrices okudlulisela phambili kuwo wonke ama-GPU ngokusebenzisa i-tensor parallelism.
Ukusebenzisa i-GPipe ukubeka izendlalelo ezihlukene zombono omkhulu noma imodeli yolimi kuma-accelerator ahlukene kuyilapho i-micro-batching ibagcina bematasa.
Injini yepayipi ye-DeepSpeed ehlukanisa imodeli yepharamitha eyizigidigidi ezingamakhulu ngezigaba ezindaweni eziningi.
Ukuhlanganisa i-tensor parallelism ngaphakathi kweseva eyodwa ye-8-GPU nokufana kwepayipi okuhlanganisa amaseva amaningi ukuqeqesha imodeli enkulu kakhulu emshinini owodwa.
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Model and Pipeline Parallelism quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Tensor Parallelism yamamodeli amakhulu
Imibuzo evame ukubuzwa
What is Model and Pipeline Parallelism?
Uma imodeli inkulu kakhulu ukuthi ingangena ku-GPU eyodwa, imodeli nokufana kwepayipi kuhlukanisa imodeli ngokwayo kuwo wonke amadivayisi. Yilokhu okwenza ukuqeqesha amamodeli olimi amakhulu anamakhulu ezigidigidi zamapharamitha angenzeka ngokomzimba.
Iyiphi inkinga ebalulekile exazululwa yimodeli nokufana kwamapayipi?
Ukufana kwemodeli namapayipi kuhlukanisa imodeli ngokwayo kuwo wonke amadivayisi, okuvumela ukuqeqeshwa kwamanethiwekhi amakhulu kakhulu kunanoma iyiphi i-GPU eyodwa.
Ngabe i-tensor (intra-layer) parallelism ihlukana kanjani isebenza?
I-tensor parallelism ihlukanisa ukubala ngaphakathi kwesendlalelo esisodwa, isibonelo ukuhlukanisa i-matrix yesisindo esikhulu ukuze i-GPU ngayinye ibale ingxenye yalokho okukhiphayo.
Liyini 'ibhamuza' ekuhambisaneni kwepayipi lokungazi?
Ekuqaleni kwesinyathelo sepayipi, izigaba ezingezansi zomfula azikho okokufaka okwamanje futhi zihlala zingenzi lutho, zimosha isikhathi se-GPU; lesi sikhala esingenzi lutho sibizwa ngebhamuza.
Ukufana kwamapayipi kulinciphisa kanjani ibhamuza?
Ukwehlukanisa inqwaba ibe amaqoqo amancane kuvumela amaqoqo amancane amaningi ukuthi athathe izigaba ezahlukene ngesikhathi esisodwa, okugcina wonke ama-GPU ematasa futhi incipha isikhathi sokungenzi lutho.
Kungani i-tensor parallelism ivamise ukugcinwa endaweni eyodwa?
I-Tensor parallelism ixhumana kaningi ukuze kuhlanganiswe imiphumela ye-matrix eyingxenye, ngakho ibekwa lapho umkhawulokudonsa we-interconnect uphakeme kakhulu, ngaphakathi kwe-node phezu kwe-NVLink.