Tensor Parallelism fun Awọn awoṣe nla
Ọna kan lati pin mathimatiki inu Layer-nẹtiwọọki nkankikan kan kọja ọpọlọpọ awọn GPUs nitorinaa awoṣe ti o tobi ju fun ẹrọ kan le tun ṣiṣẹ.
Akopọ
O ṣe pataki nitori awọn awoṣe aala ni awọn ọgọọgọrun ọkẹ àìmọye ti awọn aye ti ko si GPU kan ti o le mu tabi ṣe iṣiro ni iyara to nikan.
Jin Dive
Parallelism Tensor (ti a tun pe ni afiwe awoṣe inu-Layer) ṣapa awọn matiri iwuwo kọọkan kọja awọn GPU ju fifi gbogbo awọn fẹlẹfẹlẹ sori awọn ẹrọ lọtọ. Ninu oluyipada kan, awọn isodipupo matrix nla — awọn asọtẹlẹ akiyesi ati ifunni siwaju MLP — ti pin: fun apẹẹrẹ, matrix iwuwo akọkọ ti MLP ti pin nipasẹ awọn ọwọn ati ekeji nipasẹ awọn ori ila, nitorinaa GPU kọọkan ṣe iṣiro bibẹ kan ati idinku gbogbo-din daapọ awọn abajade. Ifarabalẹ pin si awọn ori, pẹlu GPU kọọkan ti n mu ipin kan. Nitoripe gbogbo GPU ṣe apakan ti gbogbo Layer ni nigbakannaa, tensor parallelism dinku fun-GPU iranti ati iyara soke iṣiro, sugbon o nbeere loorekoore, ga-bandwidth ibaraẹnisọrọ laarin GPUs kọọkan Layer. Ti o ni idi ti o maa n fi ara mọ laarin ipade kan ti a ti sopọ nipasẹ NVLink, ati ni idapo pelu opo gigun ti epo ati afiwera data fun ikẹkọ ti o tobi pupọ ati awọn iṣẹ ṣiṣe.
Imọ-imọ-ẹrọ
Ẹtan naa, olokiki nipasẹ Megatron-LM, n yan awọn iwọn ipin nitorina ibaraẹnisọrọ jẹ iwonba. Pipin iwe-iwe matrix MLP akọkọ-ọlọgbọn jẹ ki GPU kọọkan lo aiṣedeede ni agbegbe laisi amuṣiṣẹpọ; pipin ila-ọlọgbọn keji tumọ si pe awọn abajade kan nilo ọkan gbogbo-din si apao awọn abajade apa kan. Layer kọọkan ni bayi fa aijọju meji gbogbo-dinku (siwaju) ati meji (sẹhin). Nitoripe awọn akojọpọ wọnyi n ṣẹlẹ ni gbogbo ipele, lairi jẹ gaba lori — nitorinaa parallelism tensor n gbe lẹhin awọn ọna asopọ intra-node yara bi NVLink kuku ju awọn nẹtiwọọki inter-node lọra.
Ipa Ilana
Iye owo ati isuna
Awọn ipinnu faaji ṣe awakọ iṣẹ ati idiyele iṣẹ fun awọn ọdun.
Awọn ipinnu diẹ sii
Ẹkọ imọ-ẹrọ ṣe iranlọwọ fun awọn ẹgbẹ lati yan akopọ to tọ, kii ṣe ọkan tuntun nikan.
Iṣakoso didara
Awọn yiyan imọ-ẹrọ to dara julọ dinku awọn iṣẹlẹ igbẹkẹle ni iṣelọpọ.
Ojo iwaju ti Tensor Parallelism fun Awọn awoṣe nla
Parallelism Tensor si maa wa ni ipilẹ ṣugbọn o pọ si ni idapọ si 'parallelism 3D' (tensor + pipeline + data) ati ni idapo pẹlu afiwera alamọja fun awọn awoṣe Adapọ-ti-Amoye. Awọn ilana bii Megatron-LM, DeepSpeed, ati vLLM ṣe adaṣe adaṣe. Bi GPU interconnects (NVLink, NVSwitch) ati opitika aso gba yiyara, awọn ipade-aala iye sinmi , gbigba anfani tensor-parallel awọn ẹgbẹ. Reti isọdọkan adaṣe ijafafa ti o mu awọn iwọn shard ati awọn iwọn ẹgbẹ lati dinku ibaraẹnisọrọ fun topology iṣupọ ti a fun.
Real-World imuse
Ikẹkọ awoṣe paramita 175B nipa sisọ awọn matiri iwuwo Layer kọọkan kọja 8 GPUs ninu ipade asopọ NVLink kan nipa lilo Megatron-LM.
Nsin awoṣe iwiregbe paramita 70B ni vLLM pẹlu tensor_parallel_size = 4 nitorinaa awọn iwuwo ṣe deede kọja awọn GPU mẹrin ati dahun ni akoko gidi.
Pipin akiyesi transformer ori kọja awọn GPUs ki ẹrọ kọọkan ṣe iṣiro ipin kan, lẹhinna awọn abajade isọpọ fun Layer atẹle.
Apapọ parallelism tensor laarin awọn apa ati pipeline parallelism kọja awọn apa lati kọ awọn awoṣe paramita trillion lori awọn iṣupọ GPU nla.
Awọn ewu & Awọn ọna iṣọ
Ṣiṣepe ala-ilẹ kan le tọju awọn ailagbara eto ti o gbooro.
Awọn ohun elo amayederun ati awọn idiyele itọju nigbagbogbo ni aibikita.
Aabo ati awọn ela akiyesi le dagba bi awọn eto ṣe di eka sii.
Ilana Ilana imuse
Ṣetumo lairi, didara, ati awọn ibi-afẹde idiyele ṣaaju imuse.
Aṣepari labẹ ẹru ojulowo ati awọn ipo data.
Abojuto ohun elo fun awọn aṣiṣe, fiseete, ati ipa olumulo.
Mura ipadasẹhin pada ati awọn ipa ọna esi iṣẹlẹ ṣaaju iwọn.
Tesiwaju Ṣiṣawari
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Tensor Parallelism for Large Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Itọsọna atẹle
Awoṣe ati Pipeline Parallelism
Awọn ibeere ti a beere nigbagbogbo
Kini Isọdọtun Tensor fun Awọn awoṣe Nla?
Ọna kan lati pin mathimatiki inu Layer-nẹtiwọọki nkankikan kan kọja ọpọlọpọ awọn GPUs nitorinaa awoṣe ti o tobi ju fun ẹrọ kan le tun ṣiṣẹ. O ṣe pataki nitori awọn awoṣe aala ni awọn ọgọọgọrun ọkẹ àìmọye ti awọn aye ti ko si GPU kan ti o le mu tabi ṣe iṣiro ni iyara to nikan.
Kini parallelism tensor pin kọja awọn GPU?
Tensor (intra-Layer) parallelism shards awọn matrices iwuwo inu Layer kan, nitorinaa gbogbo GPU ṣe iṣiro apakan ti Layer kanna — yato si afiwera opo gigun ti epo, eyiti o gbe gbogbo awọn fẹlẹfẹlẹ lori oriṣiriṣi GPUs.
Ni Megatron-ara MLP pipin, bawo ni a ṣe pin awọn matiri iwuwo meji lati dinku ibaraẹnisọrọ?
Pipin matrix akọkọ nipasẹ awọn ọwọn jẹ ki GPU kọọkan lo aiṣedeede ni agbegbe; Pipin keji nipasẹ awọn ori ila tumọ si apapọ gbogbo-dinkuro awọn abajade apa kan — idinku amuṣiṣẹpọ.
Kini idi ti ifaramọ tensor maa n wa laarin ipade kan?
Layer kọọkan nfa ibaraẹnisọrọ apapọ (gbogbo-dinku), nitorina eru, ijabọ ifarabalẹ nilo awọn ọna asopọ intra-node yara bi NVLink kuku ju awọn nẹtiwọọki inter-node lọra.
Bawo ni ẹrọ ifarabalẹ ṣe deede ni afiwe labẹ isọdọkan tensor?
Awọn olori akiyesi jẹ ominira, nitorinaa wọn pin kaakiri awọn GPUs — ẹrọ kọọkan ṣe iṣiro diẹ ninu awọn ori ati awọn abajade ti wa ni idapo.
Iṣiṣẹ apapọ wo ni o ṣajọpọ awọn abajade apa kan ni afiwe tensor?
Gbogbo-din awọn akopọ awọn abajade apa kan ṣe iṣiro lori GPU kọọkan ki wọn gba lori abajade Layer; eyi n ṣẹlẹ ni ọpọlọpọ igba fun Layer ni siwaju ati sẹhin kọja.