GUIDE teknik
Lineer ak Kernel yiy def
Foofu ligneer mooy wecci fowokaayu softmax quadratic ci Transformers ak benn kaf math buy eskale ligneer ak guddaayu toppalante bi.
Déglul tegtal biiIA-nettali · ~2 min
Ci xët wii2 simili jàng
Résumé
Performer ab pexem landmark la buy xayma softmax ci ay kernel yu bari ay anam yu bari, loolu mooy tax ñu mëna jënd ay sekans yu gudd lool ci wàllu ordinatër.
Plongeur bu xóot
Transformateur standard dafay xayma poñ yi ci digganté ñaari jeton yu nekk, di njëg ak mémoire yuy màgg ak kaare guddaayig toppalante (O (n ^ 2)). Foofu ligneer bi dafay binndaat xayma bi suko defee njëg bi di màgg ci anam wu ligneer (O(n)). Xalaat bu am solo bi: softmax mooy softmax (QK ^ T) V, waaye soo wecci softmax ak kàrtu kernel phi, di nga am phi (Q) (phi (K) ^ T V). Ndax bariwaayu matrix dafay booloo, dangay njëkka xayma phi(K)^T V (matrix d-by-d bu ndaw), nga moytu matrix bu mag bi n-by-n. Performer, ci Google ci 2020, def lii ap jegewaale bu dëggu bu softmax dëgg di jëfandikoo FAVOR+ (Fàttaliku bu Gaaw Jaaraleko ci man-mani Orthogonal yu baax), di rëdd ay projection yu bari yuy tëye xayma yu kernel bi ba noppi nekk ci jàmm.
Doxal ci àdduna dëgg
Liggéeyukaay genomic wala protein yu gudd fu bàyyi xel bu mat sëkk di jeexal mémoire GPU
Résumé ci wàllu dokimaa ci kaw rapoor yu gudd lool te duñu xaaj, jëfandikoo yaxu ndigg bu nuroo ak Performer
Audio bu gudd te baax wala modeling time-series fu toppalante yi di dem ba fukki junni jéego
Wàññi njëgu inferensi ci xeetu waxtaan yu yàgg yi ci wecci yenn diisaayi softmax ak ay xeetu bàyyi xel ci ligneer
Ëlëgu bàyyi xel ci ligneer ak Kernels Performer
Foofu ligneer bu sell dafay faral di topp softmax ci kalite, moo tax barab bi dafay jaxasoo ci ay hybrid: modelu espace-state (Mamba), fowokaayu lineer bu am buntu, ak architecture yuy jaxase ay couche yu mat te bari ay couche lineaire. Bi palanteer yi di puus ci ay milioŋ ciy jeton, mekaniism lineer ak sub-quadratic dañuy gëna xëcc njëg, ba noppi ñu ngi xoolaat xeetu lineer buy baaxoo am ci inference streaming ak model ci aparey yi.
Teg ko ci jëf
Lu tax mu am solo
- Njëgg ak budget. Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
- dogal yu gëna leer. Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
- Xool kalite. Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Moytul
- Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
- Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
- Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Jéego yi ci topp
- Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
- Benchmark ci biir sargal ak done yu dëggu.
- Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
- Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Bulletin bu amul fayda
Jàngal xibaar AI bu bes
Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.
Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.
Nattal sa bopp
Take the Linear Attention and Performer Kernels quiz
Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.
Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax
Laaj yi ñuy faral di laaj
Luy kernel yiy bàyyi xel ci ligneer ak def?
Foofu ligneer mooy wecci fowokaayu softmax quadratic ci Transformers ak benn kaf math buy eskale ligneer ak guddaayu toppalante bi. Performer ab pexem landmark la buy xayma softmax ci ay kernel yu bari ay anam yu bari, loolu mooy tax ñu mëna jënd ay sekans yu gudd lool ci wàllu ordinatër.
Lan moo waral softmax biñ miin di bàyyi xel ci guddaayig toppalante bi?
Softmax dafay méngale bépp peeru jeton, defar matrix poñ n-by-n, kon njëg bi dafay màgg ni O (n ^ 2).
Ban màndarga math mooy may bàyyi xel ci ligneer mu moytu matrix n-by-n?
Ndax bariwaayu matrix dafay boole, mën nga njëkka xayma phi(K)^T V, matrix d-by-d bu ndaw, ci palaasu phi(Q)phi(K)^T.
Lan mooy mekanismu FAVOR+ bu Performer bi?
FAVOR+ dafay jëfandikoo màndarga ortogonal yu baax ngir xayma kernel softmax bi te du defar matrix biy bàyyi xel ci lépp.
Lan moo waral Performer di jëfandikoo màndarga yu baax yu bari te baña jëfandikoo màndarga trigonometrik yu njëkk ya?
Man-mani yu baax yi dañuy tax xayma kernel bi nekkul negatif, di moytu instabilite ak valeur yu baaxul yi daan sonal kàrtu man-mani sin/cos yu njëkk ya.
Lan mooy xayma jafe-jafe yi ci stil Performer ci guddaayig toppalante n?
Soo defaraat xayma bi te doo musa tabax matrix n-by-n, njëg yi dañuy yokk bu baax ak guddaayu toppalante bi.
Weyal di jàng
Gid yu jëm ci loolu
Tann nañu yeneen njiit ngir topic bii