HejuruUbuyobozi bukurikira
Icyitegererezo hamwe n'umuyoboro
Tekiniki
Ubuyobozi bwa tekiniki
Inzira yo gutandukanya imibare imbere yumurongo umwe wa neural-neteur murwego rwa GPU nyinshi kuburyo moderi nini cyane kubikoresho bimwe irashobora gukora.
Umva iki gitaboAI-byavuzwe · ~2 min
It matters because frontier models have hundreds of billions of parameters that no single GPU can hold or compute fast enough alone.
Tensor parallelism (nanone yitwa intra-layer model parallelism) igabanya matrica yuburemere bwa GPUs aho gushyira ibice byose kubikoresho bitandukanye. Muri transformateur, kugwiza matrix nini-gutegera hamwe no kugaburira-imbere MLP - bigabanijwe: kurugero, matrike ya mbere ya MLP igabanijwe ninkingi naho iyakabiri kumurongo, bityo buri GPU ibara igice kandi kimwe-kigabanya guhuza ibisubizo. Icyitonderwa kigabanijwe mumutwe, hamwe na buri GPU ikora subet. Kuberako buri GPU ikora igice cya buri cyiciro icyarimwe, tensor parallelism igabanya ububiko bwa GPU kandi yihutisha kubara, ariko bisaba itumanaho ryinshi, ryagutse cyane hagati ya GPUs buri cyiciro. Niyo mpamvu ubusanzwe ifungirwa mumurongo uhujwe na NVLink, kandi igahuzwa numuyoboro hamwe namakuru ahuza amakuru kumahugurwa manini cyane no gutanga akazi.
Guhugura icyitegererezo cya 175B mugukata matrica yuburemere bwa buri gipimo hejuru ya GPUs 8 muri node imwe ihuza NVLink ukoresheje Megatron-LM.
Gukora 70B-parameter yo kuganira muri vLLM hamwe na tensor_parallel_size = 4 bityo uburemere bukwiranye na GPU enye kandi bigasubiza mugihe nyacyo.
Gutandukanya impinduka zahinduye imitwe hejuru ya GPUs kugirango buri gikoresho kibare igice, hanyuma gihuze ibisubizo kumurongo ukurikira.
Gukomatanya tensor parallelism muri node no guhuza imiyoboro ihuriweho nu guhugura tiliyoni-ibipimo byerekana imiterere nini ya GPU.
Tensor parallelism ikomeza kuba urufatiro ariko igenda irushaho guhuzwa na '3D parallelism' (tensor + umuyoboro + data) kandi igahuzwa ninzobere ihuriweho ninzobere zivanze-zinzobere. Ibikorwa nka Megatron-LM, Umuvuduko wihuse, na vLLM byikora sharding. Mugihe GPU ihuza (NVLink, NVSwitch) hamwe nigitambara cya optique bigenda byihuta, imipaka ntarengwa irekura, bigatuma amatsinda yagutse ya tensor-parallel. Tegereza ubwenge bwikora-buringaniza butoranya ibipimo bya shard hamwe nubunini bwitsinda kugirango ugabanye itumanaho rya cluster topologiya.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Inzira yo gutandukanya imibare imbere yumurongo umwe wa neural-neteur murwego rwa GPU nyinshi kuburyo moderi nini cyane kubikoresho bimwe irashobora gukora. Nibyingenzi kuko imipaka yimipaka ifite miliyari amagana yibipimo nta GPU nimwe ishobora gufata cyangwa kubara byihuse bihagije wenyine.
Tensor (intra-layer) parallelism igabanya matrica yuburemere imbere murwego, bityo buri GPU ibara igice cyurwego rumwe - itandukanye numuyoboro uhuriweho, ushyira ibice byose kuri GPU zitandukanye.
Gutandukanya matrix yambere ukoresheje inkingi reka buri GPU ikoreshe umurongo utari muto; kugabana icya kabiri kumurongo bisobanura kimwe-byose bigabanya igiteranyo cyibisubizo igice-kugabanya guhuza.
Buri cyiciro gikurura itumanaho rusange (byose-bigabanya), kubwibyo biremereye, byihuta-byihuta byumuhanda bisaba guhuza byihuse hagati ya NVLink aho gutinda buhoro buhoro imiyoboro.
Imitwe yitondere irigenga, nuko ikwirakwizwa muri GPU-buri gikoresho kibara imitwe imwe nibisohoka hamwe.
Byose-bigabanya amafaranga igice cyibisubizo bibarwa kuri buri GPU kuburyo bahuriza kubisubizo byurwego; ibi bibaho inshuro nyinshi kumurongo imbere no gusubira inyuma.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
Icyitegererezo hamwe n'umuyoboro
Tekiniki