Daidaiton Jeri da Hankalin zobe
Daidaitawar jeri yana raba jerin dogon shigarwa guda ɗaya a cikin GPUs da yawa tare da girman alamar (lokaci), kuma Hankalin Ring yana barin waɗancan GPUs su lissafta ainihin hankali ta hanyar wucewar maɓalli / ƙimar ƙima a kusa da zobe.
Dubawa
Together they make million-token context windows feasible without any single GPU holding the whole sequence.
Zurfafa nutsewa
Daidaitaccen hankali yana buƙatar kowace tambaya don ganin kowane maɓalli/daraja, don haka ƙwaƙwalwar kunnawa tana girma tare da tsayin jeri kuma cikakken K/V dole ne ya kasance. Daidaitawar jeri yana ɓarna jeri don haka kowane GPU ya mallaki ɗimbin alamomi (da tambayoyinsu, maɓallai, ƙima). Hankalin ringi sannan yana tsara GPUs a cikin zobe mai ma'ana: kowace na'ura tana kiyaye tambayoyin gida yayin da K/V ke wucewa hop-by-hop a kusa da zoben. Yayin da kowane toshe ya zo, GPU yana ƙididdige hankali kuma yana tara sakamako ta amfani da kan layi-softmax (daidaitaccen max/ jimla dabara kamar FlashAttention). Bayan cikakken madauki, kowace tambaya ta halarci kowane maɓalli daidai, ba tare da GPU da ya taɓa adana duka K/V ba. Mahimmanci, sadarwar K/V ta zo tare da ƙididdigewa, don haka yana ƙara ƙaramin farashin agogon bango.
Fahimtar Fasaha
Hankalin zobe ya dogara da softmax na kan layi: ana iya ƙididdige hankali kan toshe-toshe yayin kiyaye iyakar gudu da mai daidaita al'ada, sannan sake ƙididdige jimlar juzu'i na farko lokacin da ƙimar girma ta bayyana. Wannan ya sa sakamakon ya zama daidai da lissafi ga cikakken hankali. Zoben yana wucewa kawai K / V tenors (ma'auni mai girma tare da toshe, ba cikakken jerin ba), kuma saboda kowane sadarwar hop ya mamaye matmul block na baya, bandwidth - ba ƙwaƙwalwar ajiya ba - ya zama abin iyakancewa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Jeri daidai gwargwado da Hankalin zobe
Daidaituwar jeri yana zama daidaitaccen horo na dogon lokaci da bayyani, yawanci haɗe tare da tensor da daidaiton bututun cikin '4D' ko '5D' daidaitaccen shimfidu. Bambance-bambancen kamar rataye ko zigzag hankali suna sake daidaita aikin da abin rufe fuska ya haifar. Yi tsammanin zoben da aka sani na topology akan NVLink da haɗin kai tare da KV-cache offloading, tura tsayin mahallin aiki zuwa ga dubun-dubatar alamu don dawo da bayanai, kode, da dogayen takardu.
Aiwatar da Gaskiyar Duniya
Horar da mahallin 1M-alama ta LLM ta hanyar rarraba kowane jeri a cikin GPUs 8 tare da Hangen Ring
Tsarin layin Megatron-LM yana rage ƙwaƙwalwar kunnawa a cikin LayerNorm da yankuna masu fita
Sarrafa gabaɗayan littafi ko babban ma'ajiyar lamba a wucewa ta gaba ɗaya ba tare da yanke ba
Haɗa Hankalin Ring tare da daidaitawar tensor don dacewa da ra'ayi mai tsayi mai tsayi akan kumburin GPU da yawa.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sequence Parallelism and Ring Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Daidaiton Tensor don Manyan Samfura
Tambayoyin da ake yawan yi
What is Sequence Parallelism and Ring Attention?
Daidaitawar jeri yana raba jerin dogon shigarwa guda ɗaya a cikin GPUs da yawa tare da girman alamar (lokaci), kuma Hankalin Ring yana barin waɗancan GPUs su lissafta ainihin hankali ta hanyar wucewar maɓalli / ƙimar ƙima a kusa da zobe. Tare suna sanya windows mahallin alamar alama miliyan mai yuwuwa ba tare da GPU guda ɗaya da ke riƙe da jerin duka ba.
Tare da wane girman jeri daidai yake raba bayanai?
Daidaitawar jeri yana ɓata jeri ɗaya a cikin GPUs tare da axis/lokaci, don haka kowace na'ura tana da ɗimbin alamomi.
Menene Hankalin Ring ke wucewa tsakanin GPUs da aka shirya cikin zobe?
Kowane GPU yana kiyaye tambayoyin gida kuma yana wuce maɓalli/daraja yana toshe hop-by-hop a kusa da zobe don haka kowace tambaya a ƙarshe tana ganin kowane maɓalli.
Wace dabara ce ke ba da damar yin lissafin toshe-by-block kuma har yanzu tana dacewa da cikakkiyar kulawa daidai?
Software na kan layi yana bin matsakaicin matsakaici da jimla, yana sake ƙididdige sakamako na ɗan lokaci kamar yadda ake buƙata, yana yin ƙididdigewa mai hikimar toshe lambobi daidai da cikakkiyar kulawa.
Me yasa Hankalin ringi baya buƙatar kowane GPU ɗaya don adana cikakken K/V?
Saboda katangar K/V suna zuwa da ƙari kuma kowane GPU yana tara hankali, babu wata na'ura da ta taɓa buƙatar maɓalli/ƙimar da aka saita a ƙwaƙwalwar ajiya lokaci ɗaya.
Ta yaya Hankalin Ring ke kiyaye sadarwa daga mamaye lokacin aiki?
Kowane sadarwar K/V na hop yana cike da matrix multiplications don toshe na yanzu, don haka lokacin canja wuri yana ɓoye a bayan ƙididdigewa.