InoteveraGaidhi rinotevera
RWKV Linear Attention
Mutauro AI
Nhungamiro yehunyanzvi
Linear Attention uye Performer Kernels
Linear tarisiro inotsiva iyo quadratic softmax kutarisisa muTransformers ine math trick inoyera mutsetse nehurefu hwekutevedzana.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Pa peji ino2 min verenga
Pfupiso
Performer is a landmark method that approximates softmax using random feature kernels, making very long sequences computationally affordable.
Kudzika Kwakadzika
Standard Transformer tarisiro inounganidza zvibodzwa pakati pezviviri zvese zvematokeni, inodhura nguva uye ndangariro inokura ine sikweya yekutevedzana kureba (O(n^2)). Linear kutarisisa inonyora zvakare komputa kuti mutengo ukure chete mutsetse (O(n)). Pfungwa yakakosha: softmax attention is softmax(QK^T)V, asi kana ukatsiva softmax nekernel feature mepu phi, unowana phi(Q)(phi(K)^T V). Nekuti kuwanda kwematrix kune mubatanidzwa, unoverengera phi(K)^T V kutanga (diki d-by-d matrix), uchinzvenga hofori n-by-n mamakisi zvachose. Mutambi, anobva ku Google muna 2020, anoita iyi fungidziro yakatendeka yechokwadi softmax uchishandisa FAVOR+ (Fast Attention Via positive Orthogonal Random features), kudhirowa zvisina tsarukano fungidziro inochengeta kernel fungidziro isina rusaruro uye yakagadzikana.
Kuitwa muNyika yeChokwadi
Kugadzira refu genomic kana mapuroteni kutevedzana uko yakazara quadratic kutarisisa kwaizopedza GPU ndangariro
Document-level muchidimbu pamusoro pemishumo yakareba kwazvo pasina chunking, uchishandisa Performer-style musana
Inoshanda-refu-fomu redhiyo kana nguva-yakatevedzana modhi uko kutevedzana kunotora makumi ezviuru zvenhanho
Kuderedza mutengo wekufungidzira mune refu-chinyorwa chat modhi nekutsiva mamwe softmax layer ane mutsara-yekutarisa akasiyana.
Ramangwana reMutsetse Wekutarisa uye Muiti Kernels
Yakachena mutsara kutarisisa kunowanzo tevera softmax pamhando, saka munda uri kutenderera pamahybrids: state-space modhi (Mamba), gated linear kutarisisa, uye zvivakwa zvinosanganisa mashoma-akazara akaturikidzana ane akawanda mitsetse. Sezvo mahwindo emamiriro ekunze anosundidzira akananga kumamiriyoni ematokeni, mutsara uye sub-quadratic masisitimu ari kuwedzera kukwezva pamutengo, uye inodzokororwa-maitiro emutsara kutarisa kuri kudzokororwa kune yakanyatso kutenderera inference uye pane-mudziyo modhi.
Kuzvishandisa
Nei zvichikosha
- Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
- Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
- Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Chenjera
- Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
- Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
- Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Matanho anotevera
- Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
- Benchmark pasi pechokwadi mutoro uye data mamiriro.
- Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
- Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Tsamba yemahara
Tora iyo yemazuva ese AI pfupiso
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Take the Linear Attention and Performer Kernels quiz
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Mibvunzo inowanzo bvunzwa
What is Linear Attention and Performer Kernels?
Linear tarisiro inotsiva iyo quadratic softmax kutarisisa muTransformers ine math trick inoyera mutsetse nehurefu hwekutevedzana. Performer inzira inofananidzira softmax uchishandisa zvisina kujairika maficha kernels, zvichiita kuti marebesheni akareba akwanise kutenga.
Nei yakajairwa softmax kutarisisa ichikwira zvisina kunaka nehurefu hwekutevedzana?
Softmax kutarisisa inoenzanisa peya yezviratidzo, kugadzira n-by-n mamakisi matrix, saka mutengo unokura seO(n^2).
Ndeipi pfuma yemasvomhu inoita kuti mutsara utarise kudzivirira iyo n-by-n matrix?
Nekuti kuwanda kwematrix kune mubatanidzwa, unogona kuverengera phi(K)^T V kutanga, diki d-by-d matrix, pachinzvimbo chekuti phi(Q)phi(K)^T.
Chii chinonzi Performer's FAVOR+ maitiro anokwana?
FAVOR + inoshandisa yakanaka orthogonal isingaverengeki maficha kufungidzira iyo exponential softmax kernel pasina kugadzira iyo yakazara yekutarisisa matrix.
Sei Muiti anoshandisa zvakanaka zvisina mwero maficha kwete ekutanga trigonometric?
Positive features inochengeta kernel fungidziro isiri-negative, ichinzvenga kusagadzikana uye hunhu husina kunaka hwaitambudza kare chivi/cos mamepu.
Ndeipi fungidziro yakaoma yePerformer-style mutsara kutarisa mukutevedzana kureba n?
Nekuronga patsva komputa uye kusambovaka iyo n-by-n matrix, inodhura zviyero zvine mutsetse nehurefu hwekutevedzana.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi