Ifarabalẹ Laini ati Awọn Kernels Oluṣe
Ifarabalẹ laini rọpo ifarabalẹ softmax kuadiratiki ni Awọn Ayirapada pẹlu ẹtan mathematiki ti o ni iwọn laini pẹlu gigun lẹsẹsẹ.
Akopọ
Performer is a landmark method that approximates softmax using random feature kernels, making very long sequences computationally affordable.
Jin Dive
Ifarabalẹ Amunawa Standard ṣe iṣiro Dimegilio laarin gbogbo bata ti awọn ami, iye owo akoko ati iranti ti o dagba pẹlu onigun mẹrin ti ipari ọkọọkan (O (n ^ 2)). Linear attention rewrites the computation so cost grows only linearly (O(n)). The key idea: softmax attention is softmax(QK^T)V, but if you replace softmax with a kernel feature map phi, you get phi(Q)(phi(K)^T V). Nitori isodipupo matrix jẹ associative, o ṣe iṣiro phi(K)^T V akọkọ (matrix d-by-d kekere kan), yago fun matrix Dimegilio nla n-by-n patapata. Oṣere, lati Google ni ọdun 2020, jẹ ki eyi jẹ isunmọ otitọ ti softmax otitọ ni lilo FAVOR+ (Ifarabalẹ Yara Nipasẹ Awọn ẹya Ainidii Orthogonal rere), yiya awọn asọtẹlẹ laileto ti o jẹ ki awọn iṣiro ekuro jẹ aiṣojusọna ati iduroṣinṣin.
Imọ-imọ-ẹrọ
FAVOR + Performer isunmọ softmax kernel exp(q.k) ni lilo awọn ẹya airotẹlẹ to dara: o maapu awọn ibeere ati awọn bọtini nipasẹ awọn asọtẹlẹ Gaussian laileto ti a we sinu ijẹẹmu, ni idaniloju awọn iwuwo akiyesi ti kii ṣe odi ati yago fun awọn iduroṣinṣin nọmba ti awọn alaroye iṣaaju. Using orthogonal random features reduces variance. Ni pataki, matrix akiyesi n-nipasẹ-n ko ni ohun elo rara, nitorinaa iranti n lọ silẹ lati kuadiratiki si laini, ti n mu awọn ilana ṣiṣe ti ẹgbẹẹgbẹrun awọn ami.
Ipa Ilana
Iye owo ati isuna
Awọn ipinnu faaji ṣe awakọ iṣẹ ati idiyele iṣẹ fun awọn ọdun.
Awọn ipinnu diẹ sii
Ẹkọ imọ-ẹrọ ṣe iranlọwọ fun awọn ẹgbẹ lati yan akopọ to tọ, kii ṣe ọkan tuntun nikan.
Iṣakoso didara
Awọn yiyan imọ-ẹrọ to dara julọ dinku awọn iṣẹlẹ igbẹkẹle ni iṣelọpọ.
Ojo iwaju ti Ifarabalẹ Laini ati Awọn Kernels Oluṣe
Ifarabalẹ laini mimọ nigbagbogbo n ṣe itọpa softmax lori didara, nitorinaa aaye naa n ṣajọpọ lori awọn arabara: awọn awoṣe aaye-ipinle (Mamba), akiyesi laini gated, ati awọn ile ayaworan ti o dapọ awọn ipele ifarabalẹ ni kikun pẹlu ọpọlọpọ awọn laini. Bi awọn window ti o tọ ti titari si awọn miliọnu awọn ami-ami, laini ati awọn ilana iha-kuadiratiki n jẹ iwunilori pupọ si idiyele, ati pe akiyesi laini ara loorekoore ni a tun wo fun itọkasi ṣiṣanwọle daradara ati awọn awoṣe ẹrọ.
Real-World imuse
Ṣiṣẹda jinomiki gigun tabi awọn ilana amuaradagba nibiti akiyesi kuadiratiki kikun yoo mu iranti GPU kuro
Akopọ ipele iwe-ipamọ lori awọn ijabọ gigun pupọ laisi gige, ni lilo ẹhin-ara Oluṣe
Olohun-fọọmu-gigun ti o munadoko tabi iṣapẹẹrẹ akoko-akoko nibiti awọn ilana ti n lọ ni ẹgbẹẹgbẹrun awọn igbesẹ
Idinku idiyele idiyele ni awọn awoṣe iwiregbe ọrọ-gigun nipa rirọpo diẹ ninu awọn fẹlẹfẹlẹ softmax pẹlu awọn iyatọ akiyesi laini
Awọn ewu & Awọn ọna iṣọ
Ṣiṣepe ala-ilẹ kan le tọju awọn ailagbara eto ti o gbooro.
Awọn ohun elo amayederun ati awọn idiyele itọju nigbagbogbo ni aibikita.
Aabo ati awọn ela akiyesi le dagba bi awọn eto ṣe di eka sii.
Ilana Ilana imuse
Ṣetumo lairi, didara, ati awọn ibi-afẹde idiyele ṣaaju imuse.
Aṣepari labẹ ẹru ojulowo ati awọn ipo data.
Abojuto ohun elo fun awọn aṣiṣe, fiseete, ati ipa olumulo.
Mura ipadasẹhin pada ati awọn ipa ọna esi iṣẹlẹ ṣaaju iwọn.
Tesiwaju Ṣiṣawari
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Linear Attention and Performer Kernels quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Itọsọna atẹle
RWKV Ifarabalẹ Laini
Awọn ibeere ti a beere nigbagbogbo
What is Linear Attention and Performer Kernels?
Ifarabalẹ laini rọpo ifarabalẹ softmax kuadiratiki ni Awọn Ayirapada pẹlu ẹtan mathematiki ti o ni iwọn laini pẹlu gigun lẹsẹsẹ. Elere jẹ ọna ala-ilẹ ti o isunmọ softmax nipa lilo awọn ekuro ẹya ara ẹrọ laileto, ṣiṣe awọn ilana gigun pupọ ni iširo.
Kí nìdí wo ni boṣewa softmax akiyesi asekale ibi pẹlu ọkọọkan ipari?
Ifarabalẹ Softmax ṣe afiwe gbogbo awọn ami ami meji, ti n ṣe agbejade matrix Dimegilio n-nipasẹ-n, nitorinaa idiyele dagba bi O (n ^ 2).
Ohun-ini mathematiki wo ni o jẹ ki akiyesi laini yago fun matrix n-nipasẹ-n?
Nitoripe isodipupo matrix jẹ alajọṣepọ, o le ṣe iṣiro phi (K)^T V akọkọ, matrix d-by-d kekere kan, dipo phi (Q) phi(K)^T.
Kini ẹrọ FAVOR+ Performer's isunmọ?
FAVOR+ nlo awọn ẹya aileto orthogonal rere lati isunmọ ekuro softmax exponential lai ṣe agbekalẹ matrix akiyesi ni kikun.
Kini idi ti Oluṣeṣe lo awọn ẹya aileto rere ju awọn trigonometric iṣaaju lọ?
Awọn ẹya to dara jẹ ki awọn iṣiro ekuro ti kii ṣe odi, yago fun awọn aisedeede ati awọn iye odi ti o fa awọn maapu ẹya ẹṣẹ tẹlẹ/cos.
Kini idiju isunmọ ti akiyesi laini ara Oluṣe ni gigun ọkọọkan n?
Nipa atunto iṣiro naa ati ki o ma ṣe kọ matrix n-nipasẹ-n, awọn iwọn iye owo laini pẹlu gigun lẹsẹsẹ.