Icyitegererezo hamwe n'umuyoboro
Iyo icyitegererezo ari kinini cyane kuburyo kidahuye na GPU imwe, icyitegererezo hamwe numuyoboro ugereranije bigabanya icyitegererezo ubwacyo mubikoresho.
Incamake
This is what makes training giant language models with hundreds of billions of parameters physically possible.
Kwibira cyane
Model parallelism igabanya icyitegererezo kimwe kuri GPU nyinshi kuburyo ntamuntu numwe ukeneye gufata uburemere bwose. Hariho uburyohe bubiri. Tensor (intra-layer) parallelism igabanya imibare imbere murwego, nko gutema matrix nini yo kugwiza muri GPU buri gice kibara igice gisohoka. Umuyoboro (inter-layer) parallelism igenera ibice bitandukanye bikurikiranye kuri GPU zitandukanye, bityo igipande cya 1 kibaho kuri GPU 0, guhagarika 2 kuri GPU 1, nibindi, hamwe nibikorwa byanyuze imbere nkumurongo winteko. Ikibazo hamwe na pipine ya naive ni 'bubble': mugihe GPU 0 ikora mugice cya mbere, epfo na ruguru GPU yicaye ubusa. Pipelining igabanya buri cyiciro mo micro-batch kugirango ibyiciro byose bigume bihuze, bitezimbere imikoreshereze.
Ubushishozi
Tensor parallelism (nko muri NVIDIA Megatron-LM) igabanya matrices yuburemere inkingi- cyangwa umurongo-ikoresha kandi ikoresha byose-kugabanya kugirango ihuze ibisubizo byigice, ikomeza itumanaho imbere muri NVLink yihuta. Imiyoboro ibangikanye (GPipe, PipeDream) igabanya icyiciro muri micro-batch zinyura mubyiciro muri gahunda itangaje, bigabanya igihe 'bubble' kidafite akamaro. Byombi bikunze gutondekwa hamwe, hamwe na tensor parallelism muri node hamwe numuyoboro uhuza ibice.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza h'icyitegererezo no guhuza imiyoboro
Imikorere igenda itangiza ikibazo kitoroshye cyo guhitamo uburyo bwo kugabana icyitegererezo mubikoresho, ukoresheje umwirondoro hamwe nishakisha kugirango uhuze compte n'itumanaho. Witegereze guhuza imbaraga za tensor, umuyoboro, hamwe namakuru abangikanye (3D parallelism), gahunda nziza ya mic-batch gahunda yo gukuraho hafi yo kuvanaho imiyoboro myinshi, hamwe nibyuma bifite imiyoboro yihuse kuburyo kugabanya igipande kimwe hejuru ya chip biba bihendutse kandi bisanzwe mubikorwa bya moderi nini nini.
Gushyira mu bikorwa Isi
Guhugura imiterere-yuburyo bwa GPT hamwe na NVIDIA Megatron-LM, igabanya buri cyerekezo cya transformateur hamwe na matrica yo kugaburira imbere muri GPU binyuze muri tensor parallelism.
Gukoresha GPipe kugirango ushire ibice bitandukanye byerekezo nini cyangwa icyerekezo cyururimi kuri moteri yihuta mugihe micro-batching ituma bahuze.
Imashini ya moteri ya DeepSpeed igabanya moderi ya miliyari-ijana-yerekana ibipimo mubice byinshi.
Gukomatanya tensor parallelism imbere ya seriveri imwe ya 8-GPU hamwe na parallelisme ihuriweho na seriveri nyinshi kugirango itoze icyitegererezo kinini cyane kumashini imwe.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Model and Pipeline Parallelism quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Tensor Iringaniza Kuri Moderi Nini
Ibibazo bikunze kubazwa
What is Model and Pipeline Parallelism?
Iyo icyitegererezo ari kinini cyane kuburyo kidahuye na GPU imwe, icyitegererezo hamwe numuyoboro ugereranije bigabanya icyitegererezo ubwacyo mubikoresho. Nibyo bituma imyitozo yindimi nini yindimi hamwe na miliyari amagana yibipimo bishoboka.
Ni ikihe kibazo cyibanze icyitegererezo nicyerekezo gikemura?
Icyitegererezo hamwe numuyoboro ugereranya ibice ubwabyo mubikoresho ubwabyo, bigafasha imyitozo yimiyoboro nini cyane kuruta GPU imwe ishobora gufata.
Nigute tensor (intra-layer) parallelism itandukanya akazi?
Tensor parallelism igabanya kubara murwego rumwe, kurugero kugabanya matrix nini kuburyo buri GPU ibara igice cyibisohoka.
Niki 'bubble' muburyo bubangikanye?
Kera muntambwe, intambwe yo hasi ntagitekerezo cyinjira kandi wicaye ubusa, guta igihe cya GPU; iki cyuho cyubusa cyitwa bubble.
Nigute imiyoboro ibangikanye igabanya ububobere?
Kugabanya icyiciro muri micro-batike ireka micro-batch nyinshi zifata ibyiciro bitandukanye icyarimwe, bigatuma GPU zose zihuze kandi zigabanya igihe cyubusa.
Ni ukubera iki uburinganire bwa tensor busanzwe bubikwa mumutwe umwe?
Tensor parallelism ivugana kenshi kugirango yongere ibisubizo bya matrix igice, bityo igashyirwa aho umurongo uhuza umurongo uri hejuru, imbere yumutwe hejuru ya NVLink.