Sliding Window Attention
Sliding window tarisiro inoganhurira tokeni imwe neimwe kuenda chete kune yakamisikidzwa-saizi nharaunda yematokeni ari padyo pane kutevedzana kwese.
Pfupiso
This cuts the quadratic cost of standard attention down to linear, making long-context models far cheaper to run.
Kudzika Kwakadzika
Kuzvitarisisa kwakajairwa kunoenzanisa chiratidzo chese nechimwe chiratidzo, saka kutevedzana kwehurefu N kunoda kuenzanisa N-squared kuenzanisa. Kutsvedza hwindo kutarisisa kunogadzirisa izvi nekupa chiratidzo chega chega hwindo rehukuru W (taura 4,096 tokens) uye kungoitira vavakidzani mukati mehwindo iro. Mutengo unokura seN nguva W pachinzvimbo cheN-squared. Zvine hutsinye, kurongedza akawanda ane mahwindo akaturikidzana kunowedzera iyo inoshanda yekugamuchira ndima: mushure meL layers, ruzivo runogona kuparadzira mhiri kanenge L nguva W tokens, senge CNN iri kukura inogamuchira munda. Mistral 7B yakasimudzira izvi nefafitera 4,096-chiratidzo kumativi makumi matatu nemaviri, ichisvika painoretical 131K-chiratidzo span. Mamodheru anowanzo kusanganisa masisitimu ane mahwindo neapo neapo akazara-yekutarisa maseru kuchengetedza marefu-renji zvinongedzo.
Technical Insight
Mune mask yekutarisisa, mubvunzo uri pachinzvimbo ndinongotenderwa kuona makiyi kubva munzvimbo ndinobvisa W kuwedzera 1 kuburikidza ini (causal kesi). Iyi sparse mask inoreva kuti KV cache inongoda ekupedzisira W tokens pane layer, ichirova ndangariro panguva yechizvarwa. Nekuti hwindo rinochinja nechiratidzo chega chega, rinopetana sechisikigo neinotenderedza buffer cache iyo inodarika zvekare zvinyorwa pane kukura nekusingaperi.
Strategic Impact
Kumhanya uye chiyero
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana.
Svika uye svika
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana.
Sarudzo dzakajeka
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora.
Ramangwana reSliding Window Attention
Dhizaini dzeHybrid ikozvino dzinosiya mashoma epasirese kana akazara-yekutarisa maseru pakati peakawanda anotsvedza-mahwindo akaturikidzana, kuenzanisa kugona nekufunga kwechokwadi kwenguva refu. Gemma 2 uye mamwe mabhuroki emuno neepasirese. Tarisira kutariswa kwehwindo kuti abatanidze nemamiriro enzvimbo-yenzvimbo, kunyura kwekutarisa, uye KV-cache compression kuitira kuti mamodheru mamodheru anobata miriyoni-zviratidzo mamiriro pasina ndangariro dzinotiza. Icho chiri kuita chivakwa chekuvaka pane chivakwa che exotic optimization.
Real-World Implementation
Mistral 7B inoshandisa 4,096-token inotsvedza hwindo mukati mematanho ayo kubata marefu ekukurudzira zvakachipa pamaGPU evatengi.
Longformer inoshandisa kutariswa kwehwindo pamwe nematinji mashoma epasirese kurongedza uye kupfupisa zvinyorwa zvemapeji akawanda.
Gemma 2 inochinjanisa mahwindo emunzvimbo anotsvedza-mahwindo ane pasi-yekutarisa maseru kuenzanisa kumhanya uye kureba kurangarira.
Rolling-buffer KV cache muvabatsiri vekutaura vanochengeta chete hwindo richangoburwa rematokeni, ichivharira ndangariro panguva yekurukurirano refu.
Njodzi & Guardrails
Chokwadi chehuroyi chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana tsvakiridzo.
Kunzwa nekukasira kunogona kugadzira mhedzisiro isingaenderane pane zvikumbiro zvakafanana.
Sensitive text data inogona kuburitswa kana zvidhiraivho zvisina kusimba.
Implementation Roadmap
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sliding Window Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Grouped-Query Attention
Mibvunzo inowanzo bvunzwa
What is Sliding Window Attention?
Sliding window tarisiro inoganhurira tokeni imwe neimwe kuenda chete kune yakamisikidzwa-saizi nharaunda yematokeni ari padyo pane kutevedzana kwese. Izvi zvinodzikisira mutengo wequadratic wekutarisisa wakajairika kusvika kune mutsara, zvichiita kuti mamodheru akareba adhure zvakanyanya kuita.
Chii chakakosha computational bhenefiti yekutsvedza kutarisa kwehwindo pamusoro pekuzvitarisira wega?
Nekudzikamisa chiratidzo chimwe nechimwe kuhwindo rakagadziriswa revavakidzani veW, mutengo unokura seN nguva W (mutsara muN) pane N-squared.
Mukugadzirwa kweMistral 7B, ruzivo runogona sei kufamba kure kupfuura imwe 4,096-token hwindo?
Kufanana neCNN, imwe neimwe layer inosundidzira ruzivo hwindo rimwe mberi, saka L maseru anopa hurefu hunoshanda hweanosvika L nguva W tokens.
Sei kutsvedza hwindo kutarisisa kunoderedza ndangariro panguva yekugadzira zvinyorwa?
Sezvo chiratidzo chinongotarisa kuhwindo razvino, kiyi yechinyakare / kukosha kwekupinda kunogona kuraswa, kazhinji kuburikidza neinotenderedza buffer cache.
Ndeipi sarudzo yekugadzira iyo modhi seGemma 2 inoshandisa padivi pemahwindo anotsvedza kuchengetedza kufunga-refu-refu?
Kusanganisa apo neapo pasi rose/kuzara-kutarisisa maseru pakati peaya emuno anochengetedza echokwadi daro rekubatanidza iro rakachena hwindo rinopera simba.
Kune causal inotsvedza hwindo rehukuru W, ndeapi makiyi anogona kubvunza pane chinzvimbo chandinopinda?
Muchiitiko chinokonzeresa mubvunzo unozviona uye W minus 1 tokens nekukurumidza pamberi payo, usambofa ma tokeni.