HejuruUbuyobozi bukurikira
Tensor Iringaniza Kuri Moderi Nini
Tekiniki
Ubuyobozi bwa tekiniki
Iyo icyitegererezo ari kinini cyane kuburyo kidahuye na GPU imwe, icyitegererezo hamwe numuyoboro ugereranije bigabanya icyitegererezo ubwacyo mubikoresho.
Umva iki gitaboAI-byavuzwe · ~2 min
Ibi nibyo bituma imyitozo yindimi nini ifite miliyari amagana yibipimo bishoboka.
Model parallelism igabanya icyitegererezo kimwe kuri GPU nyinshi kuburyo ntamuntu numwe ukeneye gufata uburemere bwose. Hariho uburyohe bubiri. Tensor (intra-layer) parallelism igabanya imibare imbere murwego, nko gutema matrix nini yo kugwiza muri GPU buri gice kibara igice gisohoka. Umuyoboro (inter-layer) parallelism igenera ibice bitandukanye bikurikiranye kuri GPU zitandukanye, bityo igipande cya 1 kibaho kuri GPU 0, guhagarika 2 kuri GPU 1, nibindi, hamwe nibikorwa byanyuze imbere nkumurongo winteko. Ikibazo hamwe na pipine ya naive ni 'bubble': mugihe GPU 0 ikora mugice cya mbere, epfo na ruguru GPU yicaye ubusa. Pipelining igabanya buri cyiciro mo micro-batch kugirango ibyiciro byose bigume bihuze, bitezimbere imikoreshereze.
Guhugura imiterere-yuburyo bwa GPT hamwe na NVIDIA Megatron-LM, igabanya buri cyerekezo cya transformateur hamwe na matrica yo kugaburira imbere muri GPU binyuze muri tensor parallelism.
Gukoresha GPipe kugirango ushire ibice bitandukanye byerekezo nini cyangwa icyerekezo cyururimi kuri moteri yihuta mugihe micro-batching ituma bahuze.
Imashini ya moteri ya DeepSpeed igabanya moderi ya miliyari-ijana-yerekana ibipimo mubice byinshi.
Gukomatanya tensor parallelism imbere ya seriveri imwe ya 8-GPU hamwe na parallelisme ihuriweho na seriveri nyinshi kugirango itoze icyitegererezo kinini cyane kumashini imwe.
Imikorere igenda itangiza ikibazo kitoroshye cyo guhitamo uburyo bwo kugabana icyitegererezo mubikoresho, ukoresheje umwirondoro hamwe nishakisha kugirango uhuze compte n'itumanaho. Witegereze guhuza imbaraga za tensor, umuyoboro, hamwe namakuru abangikanye (3D parallelism), gahunda nziza ya mic-batch gahunda yo gukuraho hafi yo kuvanaho imiyoboro myinshi, hamwe nibyuma bifite imiyoboro yihuse kuburyo kugabanya igipande kimwe hejuru ya chip biba bihendutse kandi bisanzwe mubikorwa bya moderi nini nini.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Iyo icyitegererezo ari kinini cyane kuburyo kidahuye na GPU imwe, icyitegererezo hamwe numuyoboro ugereranije bigabanya icyitegererezo ubwacyo mubikoresho. Nibyo bituma imyitozo yindimi nini yindimi hamwe na miliyari amagana yibipimo bishoboka.
Icyitegererezo hamwe numuyoboro ugereranya ibice ubwabyo mubikoresho ubwabyo, bigafasha imyitozo yimiyoboro nini cyane kuruta GPU imwe ishobora gufata.
Tensor parallelism igabanya kubara murwego rumwe, kurugero kugabanya matrix nini kuburyo buri GPU ibara igice cyibisohoka.
Kera muntambwe, intambwe yo hasi ntagitekerezo cyinjira kandi wicaye ubusa, guta igihe cya GPU; iki cyuho cyubusa cyitwa bubble.
Kugabanya icyiciro muri micro-batike ireka micro-batch nyinshi zifata ibyiciro bitandukanye icyarimwe, bigatuma GPU zose zihuze kandi zigabanya igihe cyubusa.
Tensor parallelism ivugana kenshi kugirango yongere ibisubizo bya matrix igice, bityo igashyirwa aho umurongo uhuza umurongo uri hejuru, imbere yumutwe hejuru ya NVLink.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
Tensor Iringaniza Kuri Moderi Nini
Tekiniki