Ubuyobozi bwa tekiniki

Icyitegererezo hamwe n'umuyoboro

Iyo icyitegererezo ari kinini cyane kuburyo kidahuye na GPU imwe, icyitegererezo hamwe numuyoboro ugereranije bigabanya icyitegererezo ubwacyo mubikoresho.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Ejo hazaza h'icyitegererezo no guhuza imiyoboro
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

Ibi nibyo bituma imyitozo yindimi nini ifite miliyari amagana yibipimo bishoboka.

Kwibira cyane

Model parallelism igabanya icyitegererezo kimwe kuri GPU nyinshi kuburyo ntamuntu numwe ukeneye gufata uburemere bwose. Hariho uburyohe bubiri. Tensor (intra-layer) parallelism igabanya imibare imbere murwego, nko gutema matrix nini yo kugwiza muri GPU buri gice kibara igice gisohoka. Umuyoboro (inter-layer) parallelism igenera ibice bitandukanye bikurikiranye kuri GPU zitandukanye, bityo igipande cya 1 kibaho kuri GPU 0, guhagarika 2 kuri GPU 1, nibindi, hamwe nibikorwa byanyuze imbere nkumurongo winteko. Ikibazo hamwe na pipine ya naive ni 'bubble': mugihe GPU 0 ikora mugice cya mbere, epfo na ruguru GPU yicaye ubusa. Pipelining igabanya buri cyiciro mo micro-batch kugirango ibyiciro byose bigume bihuze, bitezimbere imikoreshereze.

Gushyira mu bikorwa Isi

Guhugura imiterere-yuburyo bwa GPT hamwe na NVIDIA Megatron-LM, igabanya buri cyerekezo cya transformateur hamwe na matrica yo kugaburira imbere muri GPU binyuze muri tensor parallelism.

Gukoresha GPipe kugirango ushire ibice bitandukanye byerekezo nini cyangwa icyerekezo cyururimi kuri moteri yihuta mugihe micro-batching ituma bahuze.

Imashini ya moteri ya DeepSpeed ​​igabanya moderi ya miliyari-ijana-yerekana ibipimo mubice byinshi.

Gukomatanya tensor parallelism imbere ya seriveri imwe ya 8-GPU hamwe na parallelisme ihuriweho na seriveri nyinshi kugirango itoze icyitegererezo kinini cyane kumashini imwe.

Ejo hazaza h'icyitegererezo no guhuza imiyoboro

Imikorere igenda itangiza ikibazo kitoroshye cyo guhitamo uburyo bwo kugabana icyitegererezo mubikoresho, ukoresheje umwirondoro hamwe nishakisha kugirango uhuze compte n'itumanaho. Witegereze guhuza imbaraga za tensor, umuyoboro, hamwe namakuru abangikanye (3D parallelism), gahunda nziza ya mic-batch gahunda yo gukuraho hafi yo kuvanaho imiyoboro myinshi, hamwe nibyuma bifite imiyoboro yihuse kuburyo kugabanya igipande kimwe hejuru ya chip biba bihendutse kandi bisanzwe mubikorwa bya moderi nini nini.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Model and Pipeline Parallelism quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

Icyitegererezo nicyitegererezo ni iki?

Iyo icyitegererezo ari kinini cyane kuburyo kidahuye na GPU imwe, icyitegererezo hamwe numuyoboro ugereranije bigabanya icyitegererezo ubwacyo mubikoresho. Nibyo bituma imyitozo yindimi nini yindimi hamwe na miliyari amagana yibipimo bishoboka.

Ni ikihe kibazo cyibanze icyitegererezo nicyerekezo gikemura?

Icyitegererezo hamwe numuyoboro ugereranya ibice ubwabyo mubikoresho ubwabyo, bigafasha imyitozo yimiyoboro nini cyane kuruta GPU imwe ishobora gufata.

Nigute tensor (intra-layer) parallelism itandukanya akazi?

Tensor parallelism igabanya kubara murwego rumwe, kurugero kugabanya matrix nini kuburyo buri GPU ibara igice cyibisohoka.

Niki 'bubble' muburyo bubangikanye?

Kera muntambwe, intambwe yo hasi ntagitekerezo cyinjira kandi wicaye ubusa, guta igihe cya GPU; iki cyuho cyubusa cyitwa bubble.

Nigute imiyoboro ibangikanye igabanya ububobere?

Kugabanya icyiciro muri micro-batike ireka micro-batch nyinshi zifata ibyiciro bitandukanye icyarimwe, bigatuma GPU zose zihuze kandi zigabanya igihe cyubusa.

Ni ukubera iki uburinganire bwa tensor busanzwe bubikwa mumutwe umwe?

Tensor parallelism ivugana kenshi kugirango yongere ibisubizo bya matrix igice, bityo igashyirwa aho umurongo uhuza umurongo uri hejuru, imbere yumutwe hejuru ya NVLink.