Ukunaka Kwewindi Elishelelayo
Ukunaka kwewindi elislayidayo kuvimbela ithokheni ngayinye ukuthi ihambele kuphela indawo enosayizi ongashintshi wamathokheni aseduze esikhundleni salo lonke uchungechunge.
Uhlolojikelele
This cuts the quadratic cost of standard attention down to linear, making long-context models far cheaper to run.
I-Deep Dive
Ukuzinaka okujwayelekile kuqhathanisa yonke ithokheni nawo wonke amanye amathokheni, ngakho ukulandelana kobude N kudinga cishe ukuqhathanisa okuphindwe kabili kuka-N. Ukunaka kwewindi elislayidayo kulungisa lokhu ngokunikeza ithokheni ngayinye iwindi likasayizi W (yisho amathokheni angu-4,096) futhi unakekele kuphela omakhelwane abangaphakathi kwalelo windi. Izindleko zikhula njengo-N izikhathi W esikhundleni sika-N-squared. Ngokudabukisayo, ukupakisha izendlalelo eziningi ezinamafasitela kunweba inkambu ephumelelayo yokwamukela: ngemva kwezendlalelo ze-L, ulwazi lungasakazeka cishe kumathokheni wezikhathi ezingu-L, njengenkambu ekhulayo yokwamukela ye-CNN. I-Mistral 7B yenza lokhu kwaduma ngefasitela lamathokheni angu-4,096 kuzo zonke izendlalelo ezingu-32, lafinyelela ku-span yethokheni engu-131K. Amamodeli avame ukuhlanganisa izendlalelo ezinamafasitela kanye nezendlalelo zokunaka okugcwele ngezikhathi ezithile ukuze kugcinwe izixhumanisi zebanga elide.
I-Technical Insight
Kumaski wokunaka, umbuzo endaweni i uvunyelwe kuphela ukubona okhiye ukusuka ezindaweni ngisusa u-W kanye no-1 kuye ku-i (icala eliyimbangela). This sparse mask means the KV cache only needs the last W tokens per layer, slashing memory during generation. Ngenxa yokuthi iwindi liyashintsha ngethokheni ngayinye entsha, libhanqa ngokwemvelo nesilondolozi esigoqayo esivala okufakiwe okudala kakhulu kunokuba sikhule unomphela.
I-Strategic Impact
Isivinini nesikali
Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.
Finyelela futhi ufinyelele
Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.
Izinqumo ezicacile
Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.
Ikusasa Lokunakwa Kwewindi Elishelelayo
Imiklamo eyingxubevange manje ihlanganisa izendlalelo ezimbalwa zomhlaba noma zokunaka okugcwele phakathi kwezendlalelo eziningi zamawindi ashelelayo, ukulinganisa ukusebenza kahle nokucabanga kwangempela kwebanga elide. I-Gemma 2 namanye amabhulokhi asendaweni nawomhlaba jikelele. Lindela ukunakwa kwewindi ukuze kuhlanganiswe namamodeli we-state-space, osinki bokunaka, nokucindezelwa kwenqolobane ye-KV ukuze amamodeli asemngceleni aphathe okuqukethwe kwamathokheni ayisigidi ngaphandle kwenkumbulo ebalekayo. Iba ibhulokhi yokwakha ezenzakalelayo esikhundleni sokuba nokwenza kahle kwangaphandle.
Ukuqaliswa Komhlaba Wangempela
I-Mistral 7B isebenzisa iwindi lokushelela lamathokheni angu-4,096 kuzo zonke izendlalelo zayo ukuze isingathe imiyalo emide eshibhile kuma-GPU abathengi.
I-Longformer isebenzisa ukunakwa kwefasitela kanye namathokheni ambalwa womhlaba ukuze ahlukanise futhi afinyeze amadokhumenti anamakhasi amaningi.
I-Gemma 2 ishintshanisa izendlalelo zasendaweni zamawindi ashelelayo anezendlalelo zokunaka umhlaba wonke ukuze kulinganiswe isivinini nokukhumbula kwebanga elide.
Izilondolozi ze-KV ze-Rolling-buffer kubasizi bengxoxo zigcina kuphela iwindi lakamuva lamathokheni, elivala inkumbulo phakathi nezingxoxo ezinde.
Izingozi & Guardrails
Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.
Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.
Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.
Ukuqalisa Umhlahlandlela
Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.
Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.
Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.
Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sliding Window Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
Ukunakwa Kwemibuzo Ehlanganisiwe
Imibuzo evame ukubuzwa
What is Sliding Window Attention?
Ukunaka kwewindi elislayidayo kuvimbela ithokheni ngayinye ukuthi ihambele kuphela indawo enosayizi ongashintshi wamathokheni aseduze esikhundleni salo lonke uchungechunge. Lokhu kunciphisa izindleko ze-quadratic zokunaka okujwayelekile zehle ziye kolayini, okwenza amamodeli anomongo omude ashibhe kakhulu ukuwasebenzisa.
Iyiphi inzuzo enkulu yokubala yokuslayida ukunaka kwewindi ngaphezu kokuzinaka okuvamile?
Ngokukhawulela ithokheni ngayinye efasiteleni eligxilile lomakhelwane baka-W, izindleko zikhula njenge-N izikhathi W (umugqa ngo-N) kune-N-squared.
Ngomklamo we-Mistral 7B, ulwazi lungase luhambe kanjani kude nefasitela elilodwa lamathokheni angu-4,096?
Njenge-CNN, isendlalelo ngasinye siphusha ulwazi iwindi elilodwa liye phambili, ukuze izendlalelo ze-L zinikeze isikhathi esisebenzayo samathokheni acishe abe ngu-L.
Kungani ukunaka kwewindi elislayidayo kunciphisa inkumbulo ngesikhathi sokwenziwa kombhalo?
Njengoba ithokheni ibheka kuphela iwindi layo lakamuva, ukhiye omdala/inani elifakiwe lingalahlwa, ngokuvamile kusetshenziswa inqolobane yebhafa egoqekayo.
Yiluphi ukhetho lokuklama olusetshenziswa amamodeli afana ne-Gemma 2 eduze namafasitela ashelelayo ukuze agcine ukucabanga okude?
Ukuhlanganisa izendlalelo zomhlaba jikelele/ zokunaka okugcwele ngezikhathi ezithile phakathi kwezasendaweni kulondoloza ukuxhumana kweqiniso okude okwenza amawindi ahlanzekile abe buthaka.
Ngewindi elislayidayo eliyimbangela likasayizi W, yiziphi izikhiye ezingabuza endaweni engiyinakekelayo?
Esimeni esiyimbangela umbuzo uzibona wona kanye namathokheni we-W khipha 1 ngokushesha ngaphambi kwawo, awalokothi amathokheni esikhathi esizayo.