Amaphethini Okunakwa Okungajwayelekile
Ukunaka okuncane kwenza ama-Transformer ashibhe ngokuvumela ithokheni ngalinye ukuthi libheke ingxenye ekhethwe ngokucophelela yamanye amathokheni kunawo wonke.
Uhlolojikelele
This trades a little global reach for big savings in memory and compute on long sequences.
I-Deep Dive
Ukuzinaka okugcwele kuqhathanisa wonke amathokheni kuwo wonke amanye amathokheni, ngakho-ke izindleko zikhula ngesikwele sobude bokulandelana, okuba buhlungu kumadokhumenti amade. Ukunaka okuncane kungena esikhundleni sephethini eliminyene ngesakhiwo. Imiklamo evamile ihlanganisa ukunakwa kwefasitela (lendawo), lapho ithokheni ngayinye ibona omakhelwane abaseduze kuphela; amaphethini anemigqa noma anwetshiwe aweqela phambili ukuze afinyelele umongo okude ngenani eliphansi; kanye namathokheni omhlaba, izikhundla ezimbalwa ezikhethekile ezinakekela yonke into futhi ezibhekwayo yonke into, ezisebenza njengezikhungo zolwazi. Amamodeli afana ne-Longformer, i-BigBird, ne-Sparse Transformer ahlanganisa lezi ukuze ingqikithi yenani loxhumo likhule cishe ngokomugqa esikhundleni sama-quadratically, okuvumela okuqukethwe kwezinkulungwane kuya emashumini ezinkulungwane zamathokheni.
I-Technical Insight
Esikhundleni se-matrix yokunaka egcwele ye-N-by-N, ukunaka okuncane kubala okufakiwe okukhethiwe kuphela, ngokuvamile ukuhlanganiswa kwewindi lendawo kanye nedlanzana lamarowu namakholomu. UBigBird ukhombise ngokudumile ukuthi ukuhlanganisa ukuxhumana okungahleliwe, kwewindi, nokwembulunga yonke kugcina ukuvezwa kwethiyori kokunaka okugcwele ngenkathi kunciphisa inkimbinkimbi ukusuka ku-O(N isikwele) kuye ku-O(N). Ama-kernels asebenzayo eqa okufakiwe okumboziwe ngokuphelele kunokwenza ikhompuyutha bese kuqanda.
I-Strategic Impact
Isivinini nesikali
Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.
Finyelela futhi ufinyelele
Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.
Izinqumo ezicacile
Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.
Ikusasa Lamaphethini Okunakwa Kancane
Ukunaka okungatheni kuhlala kuphakathi kwemodeli yokuqukethwe okude, okuya ngokuya kubhangqwa nama-kernels athuthukisiwe afana ne-FlashAttention kanye nobuncane obufundiwe noma obuguqukayo obukhetha ukuthi yimaphi amathokheni okufanele uwabheke ngokokufaka ngakunye. Njengoba amafasitela omongo anwebeka afinyelela ezigidini zamathokheni, izitaki ezihlanganisiwe zixuba izendlalelo ezimbalwa, eziminyene, nezendawo yezwe. Lindela i-hardware-aware sparse kernels kanye nokunaka okusekelwe kumzila ukuze uhlale unciphisa izindleko zokufunda okokufaka okude kakhulu.
Ukuqaliswa Komhlaba Wangempela
I-longformer icubungula wonke amaphepha esayensi noma imibhalo yezomthetho ngephasi eyodwa isebenzisa iwindi elishelelayo kanye nokunaka komhlaba wonke
I-BigBird ephethe umbuzo wedokhumenti ende ephendula kanye nokulandelana kwe-genomics ngokunaka kokulinganisa komugqa
Ifinyeza umbhalo wobude bencwadi lapho ukunaka okugcwele kuzoqeda inkumbulo ye-GPU
Ukubuyisa kanye nezinhlelo zengxoxo zomongo omude asebenzisa amathokheni ehabhu yomhlaba wonke ukuhambisa ulwazi olubalulekile ezinkulungwaneni zamathokheni
Izingozi & Guardrails
Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.
Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.
Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.
Ukuqalisa Umhlahlandlela
Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.
Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.
Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.
Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sparse Attention Patterns quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Block-Sparse ne-Native Sparse Attention
Imibuzo evame ukubuzwa
What is Sparse Attention Patterns?
Ukunaka okuncane kwenza ama-Transformer ashibhe ngokuvumela ithokheni ngalinye ukuthi libheke ingxenye ekhethwe ngokucophelela yamanye amathokheni kunawo wonke. Lokhu kuhweba ngokufinyelela okuncane komhlaba wonke ukuze kugcinwe imali enkulu enkumbulweni futhi kubalwe ngokulandelana okude.
Kungani ukuzinaka okugcwele kubiza ekulandeleni okude?
Ukunaka okugcwele kuqhathanisa yonke ithokheni nawo wonke amanye amathokheni, kunikeza O(N isikwele) izindleko ngesikhathi nenkumbulo.
Kuyini ukunaka kwewindi elishelelayo (lendawo)?
Ukunaka kwendawo kukhawulela umbono wethokheni ngalinye efasiteleni eligxilile lamathokheni azungezile, kunciphisa izindleko kakhulu.
Iyini inhloso 'yezimpawu zomhlaba' ngokunaka okuncane?
Amathokheni ambalwa omhlaba axhumeka kuzo zonke izikhundla, okuvumela ulwazi lugeleze kulo lonke ukulandelana ngemali ephansi.
Iyiphi imodeli efakazele ukuthi ukuhlanganisa ukunaka okungahleliwe, kwewindi, nokwembulunga yonke kugcina ukugqama kokunaka okugcwele?
I-BigBird ibonise iphethini yayo eyingcosana iwukulinganisa kwendawo yonke yemisebenzi yokulandelana kuyilapho ikala ngomugqa.
Kulinganiselwa ukuthi inani lokuxhunywa likhula kanjani ekunakeni okumbalwa okuklanywe kahle?
Ngokukhawulela ukuxhumeka kumafasitela endawo kanye nezixhumanisi ezimbalwa zomhlaba, ukuxhumana okuphelele kukhula cishe ngokomugqa.