Nleba anya windo slide
Nleba anya mpio mmịfe na-amachibido akara ngosi ọ bụla ịga naanị na mpaghara nwere oke nke token dị nso kama usoro niile.
Nchịkọta
This cuts the quadratic cost of standard attention down to linear, making long-context models far cheaper to run.
Ime miri emi
Ọkọlọtọ nlebara anya onwe onye na-atụnyere akara ọ bụla na akara ọ bụla ọzọ, yabụ usoro ogologo N chọrọ ntụnyere N-squared. Nleba anya windo na-amị amị na-edozi nke a site n'inye akara ngosi nke ọ bụla mpio nke nha W (kwuo 4,096 akara) na ịgakwuru ndị agbata obi n'ime windo ahụ. Ọnụ ego na-eto ka N ugboro W kama N-squared. N'ụzọ dị mkpa, ikpokọta ọtụtụ windo nwere windo na-agbasawanye ogige nnabata dị irè: mgbe ọkwa L, ozi nwere ike ịgbasa gafee akara ngosi L ugboro W, dị ka ubi nnabata CNN na-eto eto. Mistral 7B mebere nke a site na windo akara ngosi 4,096 n'ofe akwa 32, na-eru ogologo oge akara ngosi 131K. Modelsdị na-agwakọta ihie windo nwere nlebara anya oge ụfọdụ iji chekwaa njikọ dị ogologo.
Nghọta nka nka
Na nkpuchi nlebara anya, a na-ahapụ ajụjụ nọ n'ọnọdụ m ka ọ hụ igodo si n'ọnọdụ m wepụrụ W gbakwunyere 1 site na i (ikpe kpatara). Ihe mkpuchi a na-adịghị ahụkebe pụtara na cache KV chọrọ naanị akara W ikpeazụ n'otu oyi akwa, na-egbutu ebe nchekwa n'oge ọgbọ. N'ihi na mpio ahụ na-eji akara ọhụrụ ọ bụla na-agbanwe, ọ na-ejikọta ya na cache nchekwa na-atụgharị nke na-edegharị ndenye ndị kacha ochie kama itolite ruo mgbe ebighị ebi.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke Nlebanya enyo na-amị amị
Nhazi ngwakọ ugbu a na-ahapụ nlebara anya ole na ole zuru ụwa ọnụ ma ọ bụ nlebara anya zuru oke n'etiti ọtụtụ windo na-amị amị, na-edozi arụmọrụ yana ezi echiche ogologo oge. Gemma 2 na ndị ọzọ na-atụgharị mpaghara na gburugburu ụwa. Na-atụ anya nleba anya mpio ka ọ ga-ejikọta ya na ụdị oghere steeti, nlebara anya, na mkpakọ KV-cache ka ụdị ndị dị n'ihu na-ejikwa ọnọdụ ndenye akara na-enweghị ebe nchekwa ọpụrụiche. Ọ na-aghọ ngọngọ ụlọ nke na-adịghị mma karịa nkwalite dị egwu.
Mmejuputa n'ezie n'ụwa
Mistral 7B na-eji 4,096-token mmịfe na-agafe n'ígwé ya iji jikwaa ogologo mkpali dị ọnụ ala na GPU ndị ahịa.
Longformer na-etinye uche na windo gbakwunyere akara ngosi zuru ụwa ọnụ ole na ole iji wepụta na chịkọta akwụkwọ nwere ọtụtụ ibe.
Gemma 2 na-eji akwa nlebara anya zuru ụwa ọnụ na-atụgharị n'ígwé windo mmịfe mpaghara iji dozie ọsọ na ncheta ogologo oge.
Mpempe akwụkwọ nchekwa KV dị na ndị enyemaka nkata na-edobe naanị mpio akara ngosi kacha ọhụrụ, na-edobe ebe nchekwa n'oge mkparịta ụka dị ogologo.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sliding Window Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nlebanya Ajụjụ agbakọtara
Ajụjụ a na-ajụkarị
What is Sliding Window Attention?
Nleba anya mpio mmịfe na-amachibido akara ngosi ọ bụla ịga naanị na mpaghara nwere oke nke token dị nso kama usoro niile. Nke a na-ebelata ọnụ ahịa quadratic nke nlebara anya ọkọlọtọ gbadaa na linear, na-eme ka ụdị ogologo okwu dị ọnụ ala karịa ka ọ na-agba.
Gịnị bụ isi uru mgbakọ na mwepụ nke nlebara anya windo na-amị amị karịa nlebara anya onwe onye ọkọlọtọ?
Site na ịmachi akara ọ bụla na windo edobere nke ndị agbata obi W, ọnụ ahịa na-eto ka N ugboro W (linear na N) karịa N-squared.
N'ime atụmatụ Mistral 7B, kedu ka ozi ga-esi ka na-agabiga otu windo akara ngosi 4,096?
Dị ka CNN, oyi akwa ọ bụla na-akpali ozi otu windo n'ihu, yabụ L n'ígwé na-enye ogologo oge dị irè nke akara ngosi L ugboro W.
Kedu ihe kpatara nlebara anya windo na-amị amị ji ebelata ebe nchekwa n'oge ọgbọ ederede?
Ebe akara ngosi na-aga naanị na mpio ya na nso nso a, enwere ike tụfuo igodo/ụru ndenye ochie, mgbe mgbe site na cache nchekwa na-atụgharị.
Kedu nhọrọ imewe ụdị dị ka Gemma 2 na-eji n'akụkụ windo ndị na-amị amị iji mee ka echiche dị ogologo?
Ịgwakọta ọkwa nlebara anya zuru ụwa ọnụ oge ụfọdụ n'etiti ndị obodo na-echekwa ezi njikọ dị anya nke na-ebelata windo dị ọcha.
Maka mpio na-amị amị nke nha W, kedu igodo nwere ike jụọ ajụjụ n'ọnọdụ m na-aga?
N'okwu nke kpatara ajụjụ a na-ahụ onwe ya na akara W wepụrụ 1 ozugbo n'ihu ya, ọ dịghị mgbe ọ bụla akara ngosi n'ọdịnihu.