Tensor Parallelism maka Model buru ibu
Ụzọ isi kewaa mgbakọ na mwepụ n'ime otu neural-network oyi akwa gafee ọtụtụ GPU ka ihe nlereanya buru ibu maka otu ngwaọrụ ka nwere ike na-agba ọsọ.
Nchịkọta
It matters because frontier models have hundreds of billions of parameters that no single GPU can hold or compute fast enough alone.
Ime miri emi
Tensor parallelism (nke a na-akpọkwa intra-layer model parallelism) na-emebi matrices dị arọ nke ọ bụla n'ofe GPU kama itinye akwa niile na ngwaọrụ dị iche iche. N'ime ihe ngbanwe, nnukwu nnukwu matrix multiplications — nlebara anya nlebara anya na ntinye MLP na-aga n'ihu — na-ekewa: dịka ọmụmaatụ, a na-ekewa matrix mbụ nke MLP site na kọlụm na nke abụọ site na ahịrị, yabụ GPU ọ bụla na-agbakọ iberi na otu mbelata niile na-ejikọta nsonaazụ. A na-ekewa nlebara anya n'ofe isi, na GPU nke ọ bụla na-ejikwa obere ihe. N'ihi na GPU ọ bụla na-eme akụkụ nke oyi akwa ọ bụla n'otu oge, tensor parallelism na-ebelata ebe nchekwa GPU ọ bụla ma na-agbakọ ọsọ ọsọ, mana ọ na-achọ nkwurịta okwu ugboro ugboro na bandwidth dị elu n'etiti GPUs oyi akwa ọ bụla. Ọ bụ ya mere a na-ejikarị emechi ya n'ime ọnụ ọnụ nke NVLink jikọtara ya na pipeline na data myirịta maka ọzụzụ buru ibu na ọrụ ije ozi.
Nghọta nka nka
Aghụghọ ahụ, nke Megatron-LM na-ewu ewu, na-ahọrọ akụkụ nkebi ka nkwurịta okwu dị ntakịrị. Ịkewaa kọlụm matriks nke mbụ MLP na-eme ka GPU ọ bụla tinye ihe na-adịghị adị na mpaghara na-enweghị mmekọrịta; ikewa nke abụọ n'ahịrị-amamihe pụtara na ihe nrụpụta chọrọ naanị otu-belata iji chịkọta arụpụtaghị akụkụ. N'ihi ya, oyi akwa ọ bụla na-ebute ihe dị ka abụọ na-ebelata (n'ihu) na abụọ (azụ). N'ihi na mkpokọta ndị a na-eme oyi akwa ọ bụla, latency na-achịkwa - yabụ tensor parallelism na-ebi n'azụ njikọ intra-node ngwa ngwa dị ka NVLink kama iji nwayọọ nwayọọ na-abanye n'ime netwọk.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke Tensor Parallelism maka nnukwu Model
Myirịta tensor na-anọgide na ntọala mana ọ na-ejikọta ya na '3D parallelism' (tensor + pipeline + data) yana jikọtara ya na myirịta ndị ọkachamara maka ụdị ngwakọta-nke ndị ọkachamara. Frameworks dị ka Megatron-LM, DeepSpeed na vLLM na-arụ ọrụ sharding. Ka GPU na-ejikọta (NVLink, NVSwitch) na akwa anya na-agba ọsọ ọsọ, oke oke ọnụ na-ada jụụ, na-enye ohere ka ndị otu tensor yiri. Na-atụ anya myikọ akpaaka ka amamihe karịa nke na-ahọrọ akụkụ shard na nha otu iji wedata nzikọrịta ozi maka ụyọkọ topology enyere.
Mmejuputa n'ezie n'ụwa
Ọzụzụ ihe atụ 175B-parameter site n'ịkọwa matrices ịdị arọ nke oyi akwa ọ bụla n'ofe 8 GPU n'otu ọnụ ejikọrọ NVLink site na iji Megatron-LM.
Na-eje ozi ụdị nkata 70B-parameter na vLLM nwere tensor_parallel_size=4 ka nha ahụ dabara n'ofe GPU anọ wee zaghachi ozugbo.
Nleba anya nkewa nke ntụgharị na-eche n'ofe GPUs ka ngwaọrụ ọ bụla gbakọọ nkeji, wee na-emekọ ihe maka oyi akwa na-esote.
Na-ejikọta myirịta tensor n'ime ọnụ ọnụ na myirịta pipeline n'ofe ọnụ iji zụọ ụdị paramita trillion na nnukwu ụyọkọ GPU.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Tensor Parallelism for Large Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nlereanya na Pipeline Parallelism
Ajụjụ a na-ajụkarị
What is Tensor Parallelism for Large Models?
Ụzọ isi kewaa mgbakọ na mwepụ n'ime otu neural-network oyi akwa gafee ọtụtụ GPU ka ihe nlereanya buru ibu maka otu ngwaọrụ ka nwere ike na-agba ọsọ. Ọ dị mkpa n'ihi na ụdị oke nwere ọtụtụ narị ijeri paramita nke ọ nweghị otu GPU nwere ike ijide ma ọ bụ gbakọọ ngwa ngwa naanị ya.
Kedu ihe myirịta tensor na-ekewa n'ofe GPUs?
Tensor (intra-layer) parallelism parallelism na-emebi matrices dị arọ n'ime akwa oyi akwa, yabụ GPU ọ bụla na-agbakọ akụkụ nke otu oyi akwa - dị iche na myirịta pipeline, nke na-etinye akwa niile na GPU dị iche iche.
Na Megatron ụdị MLP kewara, kedu ka esi ekewa matrices dị arọ abụọ iji belata nzikọrịta ozi?
Ịkewaa matriks nke mbụ site na kọlụm na-eme ka GPU ọ bụla tinye ihe na-adịghị na ya na mpaghara; ikewa nke abụọ site n'ahịrị pụtara otu ihe na-ebelata mkpokọta mpụta - na-ebelata mmekọrịta.
Kedu ihe kpatara eji edebe tensor parallelism n'ime otu ọnụ ọnụ?
Igwe oyibo ọ bụla na-ebute nzikọrịta ozi mkpokọta (na-ebelata), yabụ okporo ụzọ dị arọ, nke nwere mmetụta na-adịghị mma na-achọ njikọ intra-node ngwa ngwa dị ka NVLink karịa netwọọdụ ọnụ ọnụ nwayọ nwayọ.
Kedu ka usoro nlebara anya si ejikọtakarị n'okpuru tensor parallelism?
Ndị isi nlebara anya nọọrọ onwe ha, ya mere a na-ekesa ha n'ofe GPU - ngwaọrụ ọ bụla na-agbakọ ụfọdụ isi yana ejikọta ya.
Kedu ọrụ mkpokọta na-ejikọtakarị arụpụta akụkụ na tensor parallelism?
Nchikota niile-belata nchikota mpụtara elele anya agbakọrọ na GPU nke ọ bụla ka ha kwenye na nsonaazụ oyi akwa; nke a na-eme ọtụtụ ugboro n'otu oyi akwa n'ihu na azụ azụ.