Daidaiton Tensor don Manyan Samfura
Hanya don raba lissafi a cikin layin cibiyar sadarwa guda ɗaya a cikin GPUs da yawa don haka samfurin da ya yi girma ga na'ura ɗaya har yanzu yana iya aiki.
Dubawa
It matters because frontier models have hundreds of billions of parameters that no single GPU can hold or compute fast enough alone.
Zurfafa nutsewa
Daidaituwar Tensor (wanda kuma ake kira daidaitaccen ƙirar intra-Layer) yana ɓata ma'aunin nauyi ɗaya a cikin GPUs maimakon sanya yadudduka gabaɗaya akan na'urori daban. A cikin na'ura mai canzawa, babban matrix multiplications-hasashen hankali da kuma ciyarwar gaba MLP-an raba: misali, MLP's na farko nauyi matrix an raba shi da ginshiƙai da na biyu ta layuka, don haka kowane GPU yana lissafta yanki kuma guda ɗaya-ragu yana haɗa sakamakon. Hankali yana raba kan kawunansu, tare da kowane GPU yana sarrafa wani yanki. Saboda kowane GPU yana yin wani ɓangare na kowane Layer lokaci guda, daidaitawar tensor yana rage ƙwaƙwalwar kowane-GPU kuma yana haɓaka ƙididdigewa, amma yana buƙatar sadarwa akai-akai, babban bandwidth tsakanin GPUs kowane Layer. Shi ya sa galibi ana tsare shi a cikin kumburin da NVLink ya haɗa, kuma a haɗa shi da bututun mai da daidaiton bayanai don manyan horo da ayyukan yi.
Fahimtar Fasaha
Dabarar, wacce Megatron-LM ta shahara, tana zabar girman rabo don haka sadarwa ba ta da yawa. Rarraba ginshiƙi na farko na MLP matrix-hikima yana barin kowane GPU yayi amfani da rashin daidaituwa a cikin gida ba tare da daidaitawa ba; Rarraba layi na biyu-hikima yana nufin abubuwan da ake fitarwa kawai suna buƙatar guda ɗaya-rage don taƙaita sakamako na ɓangarori. Kowane Layer don haka yana haifar da kusan duka biyu-rage (gaba) da biyu (a baya). Saboda waɗannan ƙungiyoyin suna faruwa kowane layi, latency ya mamaye-don haka tensor parallelism yana rayuwa a bayan hanyoyin haɗin-ƙulli mai sauri kamar NVLink maimakon hanyoyin sadarwa na kuɗaɗen hankali.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Tensor Parallelism don Manyan Samfura
Daidaitawar Tensor ya kasance mai tushe amma yana ƙara haɗawa cikin 'daidaitawar 3D' (tensor + bututun + bayanai) kuma a haɗe shi da daidaiton ƙwararru don ƙirar Masana-Cura. Tsarin tsarin kamar Megatron-LM, DeepSpeed , da vLLM suna sarrafa sharding. Kamar yadda haɗin gwiwar GPU (NVLink, NVSwitch) da yadudduka na gani ke samun sauri, iyakar iyakar-ƙorafi yana shakatawa, yana ba da damar ƙungiyoyi masu daidaitawa. Yi tsammanin daidaitawa kai tsaye mai wayo wanda ke ɗaukar girman ɓangarorin da girman rukuni don rage girman sadarwa don wani nau'i na topology da aka bayar.
Aiwatar da Gaskiyar Duniya
Horar da samfurin siga na 175B ta hanyar karkatar da ma'aunin nauyi na kowane Layer a cikin 8 GPUs a cikin kumburin haɗin NVLink guda ɗaya ta amfani da Megatron-LM.
Yin hidimar ƙirar siga na 70B a cikin vLLM tare da tensor_parallel_size = 4 don haka ma'aunin ya yi daidai da GPU guda huɗu kuma yana amsawa a cikin ainihin lokaci.
Rarraba hankalin mai canzawa yana kaiwa a ko'ina cikin GPUs don haka kowace na'ura ta ƙididdige juzu'i, sannan tattara abubuwan da aka fitar don Layer na gaba.
Haɗa daidaiton tensor tsakanin nodes da daidaiton bututun mai a fadin nodes don horar da ƙira-ƙira- tiriliyan akan manyan gungu na GPU.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Tensor Parallelism for Large Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Model da Bututu Daidaita
Tambayoyin da ake yawan yi
What is Tensor Parallelism for Large Models?
Hanya don raba lissafi a cikin layin cibiyar sadarwa guda ɗaya a cikin GPUs da yawa don haka samfurin da ya yi girma ga na'ura ɗaya har yanzu yana iya aiki. Yana da mahimmanci saboda ƙirar kan iyaka suna da ɗaruruwan biliyoyin sigogi waɗanda babu GPU ɗaya da zai iya riƙe ko ƙididdigewa da sauri shi kaɗai.
Menene daidaiton tensor ya raba tsakanin GPUs?
Tensor (intra-Layer) daidaitattun ma'aunin nauyi a cikin Layer, don haka kowane GPU yana ƙididdige wani yanki na Layer iri ɗaya-bambanta da daidaiton bututun mai, wanda ke sanya yadudduka gabaɗaya akan GPUs daban-daban.
A cikin rarrabuwar MLP-style Megatron, ta yaya ake raba matrices masu nauyi biyu don rage sadarwa?
Rarraba matrix na farko ta ginshiƙai yana barin kowane GPU ya yi amfani da rashin daidaituwa a cikin gida; Rarraba na biyu ta layuka yana nufin rangwame guda ɗaya na juzu'i na abubuwan da aka samu - rage aiki tare.
Me yasa ake yawan kasancewa daidai da tensor a cikin kulli guda?
Kowane Layer yana haifar da sadarwar gama gari (duk-yana ragewa), don haka nauyi, cunkoson ababen hawa yana buƙatar hanyoyin haɗin-ƙulli mai sauri kamar NVLink maimakon a hankali hanyoyin sadarwa na kumburi.
Ta yaya tsarin kulawa yawanci ya zama daidai da daidaici ƙarƙashin tensor parallelism?
Shugabannin kulawa suna da zaman kansu, don haka ana rarraba su a cikin GPUs-kowace na'ura tana ƙididdige wasu kawunan kuma ana haɗa abubuwan da aka fitar.
Wane aiki na gama gari ne ke haɗa sakamako na ɓangarori a cikin daidaiton tensor?
Duk-rage jimlar abubuwan da aka lissafa a kan kowane GPU don su yarda da sakamakon Layer; wannan yana faruwa sau da yawa a kowane Layer a gaba da baya.