Ubuyobozi bwa tekiniki

Tensor Iringaniza Kuri Moderi Nini

Inzira yo gutandukanya imibare imbere yumurongo umwe wa neural-neteur murwego rwa GPU nyinshi kuburyo moderi nini cyane kubikoresho bimwe irashobora gukora.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Kazoza ka Tensor Kuringaniza kubintu binini
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

It matters because frontier models have hundreds of billions of parameters that no single GPU can hold or compute fast enough alone.

Kwibira cyane

Tensor parallelism (nanone yitwa intra-layer model parallelism) igabanya matrica yuburemere bwa GPUs aho gushyira ibice byose kubikoresho bitandukanye. Muri transformateur, kugwiza matrix nini-gutegera hamwe no kugaburira-imbere MLP - bigabanijwe: kurugero, matrike ya mbere ya MLP igabanijwe ninkingi naho iyakabiri kumurongo, bityo buri GPU ibara igice kandi kimwe-kigabanya guhuza ibisubizo. Icyitonderwa kigabanijwe mumutwe, hamwe na buri GPU ikora subet. Kuberako buri GPU ikora igice cya buri cyiciro icyarimwe, tensor parallelism igabanya ububiko bwa GPU kandi yihutisha kubara, ariko bisaba itumanaho ryinshi, ryagutse cyane hagati ya GPUs buri cyiciro. Niyo mpamvu ubusanzwe ifungirwa mumurongo uhujwe na NVLink, kandi igahuzwa numuyoboro hamwe namakuru ahuza amakuru kumahugurwa manini cyane no gutanga akazi.

Gushyira mu bikorwa Isi

Guhugura icyitegererezo cya 175B mugukata matrica yuburemere bwa buri gipimo hejuru ya GPUs 8 muri node imwe ihuza NVLink ukoresheje Megatron-LM.

Gukora 70B-parameter yo kuganira muri vLLM hamwe na tensor_parallel_size = 4 bityo uburemere bukwiranye na GPU enye kandi bigasubiza mugihe nyacyo.

Gutandukanya impinduka zahinduye imitwe hejuru ya GPUs kugirango buri gikoresho kibare igice, hanyuma gihuze ibisubizo kumurongo ukurikira.

Gukomatanya tensor parallelism muri node no guhuza imiyoboro ihuriweho nu guhugura tiliyoni-ibipimo byerekana imiterere nini ya GPU.

Kazoza ka Tensor Kuringaniza kubintu binini

Tensor parallelism ikomeza kuba urufatiro ariko igenda irushaho guhuzwa na '3D parallelism' (tensor + umuyoboro + data) kandi igahuzwa ninzobere ihuriweho ninzobere zivanze-zinzobere. Ibikorwa nka Megatron-LM, Umuvuduko wihuse, na vLLM byikora sharding. Mugihe GPU ihuza (NVLink, NVSwitch) hamwe nigitambara cya optique bigenda byihuta, imipaka ntarengwa irekura, bigatuma amatsinda yagutse ya tensor-parallel. Tegereza ubwenge bwikora-buringaniza butoranya ibipimo bya shard hamwe nubunini bwitsinda kugirango ugabanye itumanaho rya cluster topologiya.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Tensor Parallelism for Large Models quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is Tensor Parallelism for Large Models?

Inzira yo gutandukanya imibare imbere yumurongo umwe wa neural-neteur murwego rwa GPU nyinshi kuburyo moderi nini cyane kubikoresho bimwe irashobora gukora. Nibyingenzi kuko imipaka yimipaka ifite miliyari amagana yibipimo nta GPU nimwe ishobora gufata cyangwa kubara byihuse bihagije wenyine.

Niki tensor parallelism igabanije muri GPUs?

Tensor (intra-layer) parallelism igabanya matrica yuburemere imbere murwego, bityo buri GPU ibara igice cyurwego rumwe - itandukanye numuyoboro uhuriweho, ushyira ibice byose kuri GPU zitandukanye.

Muburyo bwa Megatron-MLP yacitsemo ibice, ni gute matrica ebyiri zuburemere zigabanijwe kugirango hagabanuke itumanaho?

Gutandukanya matrix yambere ukoresheje inkingi reka buri GPU ikoreshe umurongo utari muto; kugabana icya kabiri kumurongo bisobanura kimwe-byose bigabanya igiteranyo cyibisubizo igice-kugabanya guhuza.

Ni ukubera iki uburinganire bwa tensor busanzwe bubikwa mumutwe umwe?

Buri cyiciro gikurura itumanaho rusange (byose-bigabanya), kubwibyo biremereye, byihuta-byihuta byumuhanda bisaba guhuza byihuse hagati ya NVLink aho gutinda buhoro buhoro imiyoboro.

Nigute uburyo bwo kwitondera busanzwe bugereranywa na tensor parallelism?

Imitwe yitondere irigenga, nuko ikwirakwizwa muri GPU-buri gikoresho kibara imitwe imwe nibisohoka hamwe.

Ni ubuhe buryo rusange bukomatanya ibisubizo byigice muri tensor parallelism?

Byose-bigabanya amafaranga igice cyibisubizo bibarwa kuri buri GPU kuburyo bahuriza kubisubizo byurwego; ibi bibaho inshuro nyinshi kumurongo imbere no gusubira inyuma.