Lineer ak Kernel yiy def
Foofu ligneer mooy wecci fowokaayu softmax quadratic ci Transformers ak benn kaf math buy eskale ligneer ak guddaayu toppalante bi.
Résumé
Performer is a landmark method that approximates softmax using random feature kernels, making very long sequences computationally affordable.
Plongeur bu xóot
Transformateur standard dafay xayma poñ yi ci digganté ñaari jeton yu nekk, di njëg ak mémoire yuy màgg ak kaare guddaayig toppalante (O (n ^ 2)). Foofu ligneer bi dafay binndaat xayma bi suko defee njëg bi di màgg ci anam wu ligneer (O(n)). Xalaat bu am solo bi: softmax mooy softmax (QK ^ T) V, waaye soo wecci softmax ak kàrtu kernel phi, di nga am phi (Q) (phi (K) ^ T V). Ndax bariwaayu matrix dafay booloo, dangay njëkka xayma phi(K)^T V (matrix d-by-d bu ndaw), nga moytu matrix bu mag bi n-by-n. Performer, ci Google ci 2020, def lii ap jegewaale bu dëggu bu softmax dëgg di jëfandikoo FAVOR+ (Fàttaliku bu Gaaw Jaaraleko ci man-mani Orthogonal yu baax), di rëdd ay projection yu bari yuy tëye xayma yu kernel bi ba noppi nekk ci jàmm.
Gis-gis xarala
FAVOR+ Performer dafay xayma kernel softmax exp(q.k) ci jëfandikoo ay màndarga yu bari yu baax: dafay xayma laaj yi ak caabi yi jaaraleko ci ay projection Gaussian yu bari yuñ laxas ci exponentiel, loolu dafay garanti poid yu baaxul yi ak moytu instabilite numérique yu njëkk yi. Jëfandikoo màndarga yu bari yu ortogonal dafay wàññi variance. Li gëna am solo mooy matrix n-by-n musul am, moo tax mémoire bi dafay wàcci ci quadratic dem ci linear, loolu mooy tax ñu mëna toppalante ay fukki junni token.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu bàyyi xel ci ligneer ak Kernels Performer
Foofu ligneer bu sell dafay faral di topp softmax ci kalite, moo tax barab bi dafay jaxasoo ci ay hybrid: modelu espace-state (Mamba), fowokaayu lineer bu am buntu, ak architecture yuy jaxase ay couche yu mat te bari ay couche lineaire. Bi palanteer yi di puus ci ay milioŋ ciy jeton, mekaniism lineer ak sub-quadratic dañuy gëna xëcc njëg, ba noppi ñu ngi xoolaat xeetu lineer buy baaxoo am ci inference streaming ak model ci aparey yi.
Doxal ci àdduna dëgg
Liggéeyukaay genomic wala protein yu gudd fu bàyyi xel bu mat sëkk di jeexal mémoire GPU
Résumé ci wàllu dokimaa ci kaw rapoor yu gudd lool te duñu xaaj, jëfandikoo yaxu ndigg bu nuroo ak Performer
Audio bu gudd te baax wala modeling time-series fu toppalante yi di dem ba fukki junni jéego
Wàññi njëgu inferensi ci xeetu waxtaan yu yàgg yi ci wecci yenn diisaayi softmax ak ay xeetu bàyyi xel ci ligneer
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Linear Attention and Performer Kernels quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
RWKV Lineer
Laaj yi ñuy faral di laaj
What is Linear Attention and Performer Kernels?
Foofu ligneer mooy wecci fowokaayu softmax quadratic ci Transformers ak benn kaf math buy eskale ligneer ak guddaayu toppalante bi. Performer ab pexem landmark la buy xayma softmax ci ay kernel yu bari ay anam yu bari, loolu mooy tax ñu mëna jënd ay sekans yu gudd lool ci wàllu ordinatër.
Lan moo waral softmax biñ miin di bàyyi xel ci guddaayig toppalante bi?
Softmax dafay méngale bépp peeru jeton, defar matrix poñ n-by-n, kon njëg bi dafay màgg ni O (n ^ 2).
Ban màndarga math mooy may bàyyi xel ci ligneer mu moytu matrix n-by-n?
Ndax bariwaayu matrix dafay boole, mën nga njëkka xayma phi(K)^T V, matrix d-by-d bu ndaw, ci palaasu phi(Q)phi(K)^T.
Lan mooy mekanismu FAVOR+ bu Performer bi?
FAVOR+ dafay jëfandikoo màndarga ortogonal yu baax ngir xayma kernel softmax bi te du defar matrix biy bàyyi xel ci lépp.
Lan moo waral Performer di jëfandikoo màndarga yu baax yu bari te baña jëfandikoo màndarga trigonometrik yu njëkk ya?
Man-mani yu baax yi dañuy tax xayma kernel bi nekkul negatif, di moytu instabilite ak valeur yu baaxul yi daan sonal kàrtu man-mani sin/cos yu njëkk ya.
Lan mooy xayma jafe-jafe yi ci stil Performer ci guddaayig toppalante n?
Soo defaraat xayma bi te doo musa tabax matrix n-by-n, njëg yi dañuy yokk bu baax ak guddaayu toppalante bi.