Hankalin Tagar Zamiya
Hankalin taga mai zamewa yana ƙuntata kowace alama don halarta kawai zuwa ƙayyadadden ƙayyadadden yanki na alamun da ke kusa a maimakon duka jeri.
Dubawa
This cuts the quadratic cost of standard attention down to linear, making long-context models far cheaper to run.
Zurfafa nutsewa
Daidaitaccen kulawar kai yana kwatanta kowane alama da kowace alama, don haka jerin tsayin N yana buƙatar kwatancen N-squared. Hankalin taga mai zamewa yana gyara wannan ta hanyar baiwa kowane alamar tagar girman W (ka ce alamun 4,096) kuma kawai halartar maƙwabta a cikin wannan taga. Farashin yana girma azaman N sau W maimakon N-squared. Mahimmanci, tara yadudduka masu tagar taga da yawa yana faɗaɗa ingantaccen filin karɓa: bayan L yadudduka, bayanai na iya yaɗuwa cikin kusan alamun L sau W, kamar filin karɓar karɓar CNN. Mistral 7B ya yada wannan tare da taga mai alamar 4,096 a cikin yadudduka 32, ya kai madaidaicin 131K-token. Samfura sukan haɗa yadudduka masu taga tare da cikakken kulawa lokaci-lokaci don adana hanyoyin haɗin kai mai tsayi.
Fahimtar Fasaha
A cikin abin rufe fuska, tambaya a matsayi na kawai ana ba da izinin ganin maɓallai daga matsayi na rage W da 1 ta hanyar i (harlin dalili). Wannan abin rufe fuska yana nufin ma'ajin KV kawai yana buƙatar alamun W na ƙarshe a kowane Layer, yanke ƙwaƙwalwar ajiya yayin tsarawa. Saboda taga yana canzawa tare da kowace sabuwar alama, tana haɗe-haɗe ta halitta tare da cache buffer mai juyi wanda ke sake rubuta tsoffin shigarwar maimakon girma har abada.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Hankalin Tagar Zamiya
Haɓaka ƙira yanzu suna ba da ƴan yadudduka na duniya ko cikakkiyar kulawa a tsakanin yadudduka masu zamewa-taga, daidaita inganci tare da tunani mai tsayi na gaske. Gemma 2 da sauransu suna canza shinge na gida da na duniya. Yi tsammanin kulawar taga don haɗawa tare da ƙirar sararin samaniya, kulawar hankali, da matsawar KV-cache don haka samfuran kan iyaka suna ɗaukar mahallin alama miliyan ba tare da ƙwaƙwalwar gudu ba. Yana zama tsoho tubalin ginin maimakon ingantaccen haɓakawa.
Aiwatar da Gaskiyar Duniya
Mistral 7B yana amfani da taga mai zamewa mai alamar 4,096 a cikin yadudduka don ɗaukar dogon lokaci da rahusa akan GPUs masu amfani.
Longformer yana amfani da hankalin taga tare da ƴan alamun duniya don rarrabewa da taƙaita takaddun shafuka masu yawa.
Gemma.
KV mai buffer-buffer caches a cikin mataimakan taɗi suna adana tagar kwanan nan na alamun kawai, yana ɗaukar ƙwaƙwalwar ajiya yayin doguwar tattaunawa.
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sliding Window Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Hankalin Tambaya-Rukuni
Tambayoyin da ake yawan yi
What is Sliding Window Attention?
Hankalin taga mai zamewa yana ƙuntata kowace alama don halarta kawai zuwa ƙayyadadden ƙayyadadden yanki na alamun da ke kusa a maimakon duka jeri. Wannan yana rage ƙimar ƙima na daidaitattun hankali zuwa layi, yana mai da samfuran yanayi mai tsayi mai arha don gudu.
Menene babban fa'idar lissafi na zamewar hankalin taga akan daidaitaccen kulawar kai?
Ta iyakance kowane alamar zuwa ƙayyadadden taga na maƙwabta W, farashi yana girma kamar N sau W (mai layi a cikin N) maimakon N-squared.
A cikin ƙirar Mistral 7B, ta yaya bayanai har yanzu za su yi tafiya mai nisa fiye da taga alama guda 4,096?
Kamar CNN, kowane Layer yana tura bayanai tagar gaba ɗaya, don haka L yadudduka suna ba da ingantaccen tazara na kusan L sau W.
Me yasa hankalin taga zamewa yana rage ƙwaƙwalwar ajiya yayin tsara rubutu?
Tun da alama kawai ke zuwa taga ta kwanan nan, ana iya watsar da tsofaffin maɓalli/ƙimar shigarwar, sau da yawa ta hanyar ma'ajiyar buffer.
Wane zaɓin ƙirar ƙira irin su Gemma 2 ke amfani da shi tare da tagogi masu zamewa don kiyaye dogon tunani?
Haɗa yadudduka na lokaci-lokaci na duniya/cikakken hankali tsakanin na gida yana kiyaye haɗin kai mai nisa na gaske wanda tsantsar taga yana raunana.
Don taga mai zamiya mai girman W, waɗanne maɓallai ne za a iya tambaya a matsayin da na halarta?
A cikin yanayin dalili tambayar tana ganin kanta da W dena alamun 1 nan da nan a gabanta, ba alamun gaba ba.