Tensor Iringaniza Kuri Moderi Nini
Inzira yo gutandukanya imibare imbere yumurongo umwe wa neural-neteur murwego rwa GPU nyinshi kuburyo moderi nini cyane kubikoresho bimwe irashobora gukora.
Incamake
It matters because frontier models have hundreds of billions of parameters that no single GPU can hold or compute fast enough alone.
Kwibira cyane
Tensor parallelism (nanone yitwa intra-layer model parallelism) igabanya matrica yuburemere bwa GPUs aho gushyira ibice byose kubikoresho bitandukanye. Muri transformateur, kugwiza matrix nini-gutegera hamwe no kugaburira-imbere MLP - bigabanijwe: kurugero, matrike ya mbere ya MLP igabanijwe ninkingi naho iyakabiri kumurongo, bityo buri GPU ibara igice kandi kimwe-kigabanya guhuza ibisubizo. Icyitonderwa kigabanijwe mumutwe, hamwe na buri GPU ikora subet. Kuberako buri GPU ikora igice cya buri cyiciro icyarimwe, tensor parallelism igabanya ububiko bwa GPU kandi yihutisha kubara, ariko bisaba itumanaho ryinshi, ryagutse cyane hagati ya GPUs buri cyiciro. Niyo mpamvu ubusanzwe ifungirwa mumurongo uhujwe na NVLink, kandi igahuzwa numuyoboro hamwe namakuru ahuza amakuru kumahugurwa manini cyane no gutanga akazi.
Ubushishozi
Amayeri, akunzwe na Megatron-LM, ahitamo ibipimo by'amacakubiri kuburyo itumanaho ari rito. Gutandukanya materix ya mbere ya MLP inkingi-nziza ireka buri GPU ikoreshe umurongo utari muto mugihe nta sync; kugabana umurongo wa kabiri-ubwenge bisobanura ibisubizo bikeneye kimwe gusa-kugabanya kugirango ubone ibisubizo igice. Buri cyiciro rero kigera hafi kuri byose-bigabanya (imbere) na bibiri (inyuma). Kuberako ayo matsinda abaho murwego rwose, ubukererwe bwiganje - kubwibyo tensor parallelism ibaho inyuma yihuta ihuza imiyoboro nka NVLink aho kuba buhoro buhoro imiyoboro ihuza imiyoboro.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Kazoza ka Tensor Kuringaniza kubintu binini
Tensor parallelism ikomeza kuba urufatiro ariko igenda irushaho guhuzwa na '3D parallelism' (tensor + umuyoboro + data) kandi igahuzwa ninzobere ihuriweho ninzobere zivanze-zinzobere. Ibikorwa nka Megatron-LM, Umuvuduko wihuse, na vLLM byikora sharding. Mugihe GPU ihuza (NVLink, NVSwitch) hamwe nigitambara cya optique bigenda byihuta, imipaka ntarengwa irekura, bigatuma amatsinda yagutse ya tensor-parallel. Tegereza ubwenge bwikora-buringaniza butoranya ibipimo bya shard hamwe nubunini bwitsinda kugirango ugabanye itumanaho rya cluster topologiya.
Gushyira mu bikorwa Isi
Guhugura icyitegererezo cya 175B mugukata matrica yuburemere bwa buri gipimo hejuru ya GPUs 8 muri node imwe ihuza NVLink ukoresheje Megatron-LM.
Gukora 70B-parameter yo kuganira muri vLLM hamwe na tensor_parallel_size = 4 bityo uburemere bukwiranye na GPU enye kandi bigasubiza mugihe nyacyo.
Gutandukanya impinduka zahinduye imitwe hejuru ya GPUs kugirango buri gikoresho kibare igice, hanyuma gihuze ibisubizo kumurongo ukurikira.
Gukomatanya tensor parallelism muri node no guhuza imiyoboro ihuriweho nu guhugura tiliyoni-ibipimo byerekana imiterere nini ya GPU.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Tensor Parallelism for Large Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Icyitegererezo hamwe n'umuyoboro
Ibibazo bikunze kubazwa
What is Tensor Parallelism for Large Models?
Inzira yo gutandukanya imibare imbere yumurongo umwe wa neural-neteur murwego rwa GPU nyinshi kuburyo moderi nini cyane kubikoresho bimwe irashobora gukora. Nibyingenzi kuko imipaka yimipaka ifite miliyari amagana yibipimo nta GPU nimwe ishobora gufata cyangwa kubara byihuse bihagije wenyine.
Niki tensor parallelism igabanije muri GPUs?
Tensor (intra-layer) parallelism igabanya matrica yuburemere imbere murwego, bityo buri GPU ibara igice cyurwego rumwe - itandukanye numuyoboro uhuriweho, ushyira ibice byose kuri GPU zitandukanye.
Muburyo bwa Megatron-MLP yacitsemo ibice, ni gute matrica ebyiri zuburemere zigabanijwe kugirango hagabanuke itumanaho?
Gutandukanya matrix yambere ukoresheje inkingi reka buri GPU ikoreshe umurongo utari muto; kugabana icya kabiri kumurongo bisobanura kimwe-byose bigabanya igiteranyo cyibisubizo igice-kugabanya guhuza.
Ni ukubera iki uburinganire bwa tensor busanzwe bubikwa mumutwe umwe?
Buri cyiciro gikurura itumanaho rusange (byose-bigabanya), kubwibyo biremereye, byihuta-byihuta byumuhanda bisaba guhuza byihuse hagati ya NVLink aho gutinda buhoro buhoro imiyoboro.
Nigute uburyo bwo kwitondera busanzwe bugereranywa na tensor parallelism?
Imitwe yitondere irigenga, nuko ikwirakwizwa muri GPU-buri gikoresho kibara imitwe imwe nibisohoka hamwe.
Ni ubuhe buryo rusange bukomatanya ibisubizo byigice muri tensor parallelism?
Byose-bigabanya amafaranga igice cyibisubizo bibarwa kuri buri GPU kuburyo bahuriza kubisubizo byurwego; ibi bibaho inshuro nyinshi kumurongo imbere no gusubira inyuma.