Ukunakwa Komugqa kanye Nezinhlamvu Zomdlali
Ukunakwa komugqa kungena esikhundleni sokunaka kwe-quadratic softmax kuma-Transformers ngeqhinga lezibalo elikala ngokulandelana ngobude bokulandelana.
Uhlolojikelele
Performer is a landmark method that approximates softmax using random feature kernels, making very long sequences computationally affordable.
I-Deep Dive
Ukunaka kwe-Standard Transformer kubala amaphuzu phakathi kwepheya ngalinye lamathokheni, isikhathi esibizayo nenkumbulo ekhula ngesikwele sobude bokulandelana (O(n^2)). Ukunaka komugqa kubhala kabusha ukubala ukuze izindleko zikhule ngomugqa kuphela (O(n)). Umbono oyinhloko: ukunakwa kwe-softmax yi-softmax(QK^T)V, kodwa uma ushintsha i-softmax ufake imephu yesici se-kernel phi, uthola i-phi(Q)(phi(K)^T V). Ngenxa yokuthi ukuphindaphinda kwe-matrix kuyahambisana, ubala i-phi(K)^T V kuqala (i-matrix encane ka-d-by-d), ugwema i-matrix enkulu yesikolo sika-n-by-n ngokuphelele. Umdlali, ovela ku-Google ngo-2020, wenza lokhu ukulinganisa okuthembekile kwe-softmax yangempela esebenzisa i-FAVOR+ (Ukunaka Okusheshayo Ngezici ezinhle ze-Orthogonal Random), ukudweba ukuqagela okungahleliwe okugcina izilinganiso ze-kernel zingachemile futhi zizinzile.
I-Technical Insight
I-FAVOR+ Yomdlali ilinganiselwa ku-softmax kernel exp(q.k) isebenzisa izici ezingahleliwe ezinhle: imephu imibuzo nokhiye ngokuqagela okungahleliwe kwe-Gaussian esongwe nge-exponential, iqinisekisa izisindo zokunaka ezingezona ezimbi futhi igwema ukuntengantenga kwezinombolo kwezilinganiso zangaphambilini. Ukusebenzisa izici ezingahleliwe ze-orthogonal kunciphisa ukuhluka. Ngokudabukisayo, i-matrix yokunaka ye-n-by-n ayikaze yenzeke, ngakho inkumbulo iyehla isuka ku-quadratic iye komugqa, ivumela ukulandelana kwamashumi ezinkulungwane zamathokheni.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Lokunaka Okuqondile kanye Nezinhlamvu Zomdlali
Ukunakwa komugqa okumsulwa kuvame ukulandela i-softmax kukhwalithi, ngakho-ke inkambu ihlangana kuma-hybrids: amamodeli e-state-space (i-Mamba), ukunaka komugqa ofakwe isango, nezakhiwo ezihlanganisa izendlalelo ezimbalwa zokunaka okugcwele neziningi ezinomugqa. Njengoba umongo amawindi ephushela ezigidini zamathokheni, izindlela zomugqa kanye ne-sub-quadratic ziya ngokuya zikhanga ngezindleko, futhi ukunaka komugqa wesitayela esiphindaphindiwe kuyabuyekezwa ukuze kutholakale inkomba yokusakaza ephumelelayo namamodeli akudivayisi.
Ukuqaliswa Komhlaba Wangempela
Ukucubungula ukulandelana okude kwe-genomic noma amaprotheni lapho ukunaka okugcwele kwe-quadratic kuzoqeda inkumbulo ye-GPU
Ukufingqwa kwezinga ledokhumenti ngemibiko emide kakhulu ngaphandle kokuhlanganisa, kusetshenziswa umgogodla wesitayela soMsebenzi
Ukumodela okusebenzayo komsindo wefomu ende noma uchungechunge lwesikhathi lapho ukulandelana kuthatha amashumi ezinkulungwane zezinyathelo
Ukunciphisa izindleko zokucabanga kumamodeli engxoxo yomongo omude ngokushintsha izendlalelo ezithile ze-softmax ngokuhlukahluka kokunakwa komugqa
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Linear Attention and Performer Kernels quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-RWKV Linear Attention
Imibuzo evame ukubuzwa
What is Linear Attention and Performer Kernels?
Ukunakwa komugqa kungena esikhundleni sokunaka kwe-quadratic softmax kuma-Transformers ngeqhinga lezibalo elikala ngokulandelana ngobude bokulandelana. I-Performer iyindlela eyingqopha-mlando ecishe ifane ne-softmax isebenzisa izici ezingahleliwe, okwenza ukulandelana okude kakhulu kufinyeleleke ngokwekhompiyutha.
Kungani ukunaka okujwayelekile kwe-softmax kukala kabi ngobude bokulandelana?
Ukunaka kwe-Softmax kuqhathanisa wonke amathokheni, okukhiqiza i-matrix yesikolo sika-n-by-n, ngakho izindleko zikhula njengo-O(n^2).
Iyiphi indawo yezibalo evumela ukunakwa komugqa kugweme i-matrix ye-n-by-n?
Ngenxa yokuthi ukuphindaphinda kwe-matrix kuyahlanganisa, ungakwazi ukubala okuthi phi(K)^T V kuqala, i-matrix encane ka-d-by-d, esikhundleni sokuthi phi(Q)phi(K)^T.
Ilinganisa ini indlela ye-Performer's FAVOR+?
I-FAVOR+ isebenzisa izici ezingahleliwe ezinhle ze-orthogonal ukulinganisa i-exponential softmax kernel ngaphandle kokwenza i-matrix yokunaka egcwele.
Kungani uMenzi esebenzisa izici ezinhle ezingahleliwe kunezangaphambili ze-trigonometric?
Izici ezinhle zigcina izilinganiso ze-kernel zingezimbi, zigwema ukuntengantenga kanye namanani angemahle ahlasele amamephu wesici sangaphambilini se-sin/cos.
Ingakanani inkimbinkimbi elinganiselwe yokunaka komugqa wesitayela Somdlali ngobude bokulandelana kuka-n?
Ngokuhlela kabusha ukubala futhi ungalokothi wakhe i-matrix ye-n-by-n, biza izikali ngokulandelana ngobude bokulandelana.