Imirongo Yitondewe hamwe nudukoryo
Icyerekezo cyumurongo gisimbuza quadratic softmax kwitondera muri Transformers hamwe namayeri yimibare ipima umurongo hamwe nuburebure bwikurikiranya.
Incamake
Performer is a landmark method that approximates softmax using random feature kernels, making very long sequences computationally affordable.
Kwibira cyane
Icyerekezo gisanzwe gihindura amanota hagati ya buri kimenyetso, gitwara igihe hamwe nibuka bikura hamwe na kare yuburebure (O (n ^ 2)). Kwitonda kumurongo byongeye kubara kuburyo igiciro gikura gusa umurongo (O (n)). Igitekerezo cyingenzi: softmax kwitondera ni softmax (QK ^ T) V, ariko niba usimbuye softmax hamwe nikarita yerekana ikarita phi, ubona phi (Q) (phi (K) ^ T V). Kuberako kugwiza matrix bifitanye isano, ubara phi (K) ^ T V ubanza (mato mato mato mato mato), ukirinda amanota manini n-by-n amanota yose. Abakora, guhera kuri Google muri 2020, bituma ibi bigereranywa kwizerwa rya softmax yukuri ukoresheje FAVOR + (Byihuta Byihutirwa Binyuze muburyo bwiza bwa Orthogonal Random ibiranga), ushushanya ibipimo bitunguranye bituma igereranya ryintangiriro ritabogamye kandi rihamye.
Ubushishozi
UMUKUNZI WA FAVOR + agereranya softmax kernel exp (q.k) akoresheje ibintu byiza bidasanzwe: ishushanya ibibazo nurufunguzo binyuze muri progaramu ya Gaussiya yapfunditswe mu buryo bworoshye, byemeza uburemere butari bwiza kandi birinda ihungabana ry’imibare yabateganijwe mbere. Gukoresha imiterere ya orthogonal idasanzwe igabanya itandukaniro. Icy'ingenzi, matrike ya n-by-n ntabwo yigeze iba iy'umubiri, bityo kwibuka bigabanuka kuva kuri quadratic kugera kumurongo, bigatuma urutonde rwibihumbi mirongo.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza h'umurongo wo kwitondera no gukora intangiriro
Kwitonda kumurongo ukunze gukurikira softmax kubwiza, bityo umurima ugahuza kuri Hybride: imiterere-yimiterere ya leta (Mamba), umurongo werekeza kumurongo, hamwe nubwubatsi buvanga ibice bike byuzuye-byuzuye hamwe numurongo umwe. Mugihe imiterere ya Windows isunika kuri miriyoni yikimenyetso, umurongo hamwe na sub-quadratic uburyo bugenda bushishikaza kubiciro, kandi uburyo-bwo-buryo-bwo-buryo-bwo-buryo-bwo-buryo-busubirwamo burimo gusubirwamo kugirango habeho gufata neza no kwerekana ibikoresho.
Gushyira mu bikorwa Isi
Gutunganya genomic cyangwa proteine ndende aho kwitabwaho kwadratic byuzuye byananiza ububiko bwa GPU
Urwego-urwego rwincamake hejuru ya raporo ndende cyane nta gucogora, ukoresheje umugozi-wuburyo bwumugongo
Uburyo bwiza-buringaniye bwamajwi cyangwa ibihe-byerekana urugero aho urukurikirane rugera ku bihumbi mirongo
Kugabanya ibiciro byimyanzuro mugihe kirekire-biganiro biganirwaho mugusimbuza ibice bimwe bya softmax hamwe numurongo-wo kwitondera ibintu
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Linear Attention and Performer Kernels quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
RWKV Itondekanya
Ibibazo bikunze kubazwa
What is Linear Attention and Performer Kernels?
Icyerekezo cyumurongo gisimbuza quadratic softmax kwitondera muri Transformers hamwe namayeri yimibare ipima umurongo hamwe nuburebure bwikurikiranya. Abakora nuburyo bwingenzi bugereranya softmax ukoresheje intungamubiri zidasanzwe, gukora urutonde rurerure cyane kubara.
Ni ukubera iki softmax isanzwe yitabwaho cyane hamwe n'uburebure bukurikirana?
Softmax kwitondera igereranya buri jambo ryibimenyetso, ritanga amanota ya n-by-n, bityo igiciro kiyongera nka O (n ^ 2).
Ni ubuhe buryo bw'imibare butuma umurongo witondera kwirinda n-by-n matrix?
Kuberako kugwiza matrix bifitanye isano, urashobora kubara phi (K) ^ T V ubanza, mato mato mato mato, aho kuba phi (Q) phi (K) ^ T.
Ni ubuhe buryo bwa FAVOR + bukora neza?
FAVOR + ikoresha uburyo bwiza bwa orthogonal idasanzwe kugirango igereranye intungamubiri ya softmax itabanje gukora matrix yuzuye.
Kuki Performer ikoresha ibintu byiza bidasanzwe aho gukoresha trigonometric yambere?
Ibintu byiza biranga intangiriro igereranya ibitari byiza, wirinda ihungabana nagaciro keza byagaragaye mbere yikarita / cos ikarita.
Ni ubuhe buryo bugereranijwe bwa Performer-stil umurongo utondekanya muburyo bukurikiranye n?
Mugutondekanya kubara kandi ntuzigere wubaka matrike ya n-by-n, umunzani wigiciro ugereranije nuburebure bwikurikiranya.