Ubuyobozi bwa tekiniki

Imirongo Yitondewe hamwe nudukoryo

Icyerekezo cyumurongo gisimbuza quadratic softmax kwitondera muri Transformers hamwe namayeri yimibare ipima umurongo hamwe nuburebure bwikurikiranya.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Ejo hazaza h'umurongo wo kwitondera no gukora intangiriro
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

Performer is a landmark method that approximates softmax using random feature kernels, making very long sequences computationally affordable.

Kwibira cyane

Icyerekezo gisanzwe gihindura amanota hagati ya buri kimenyetso, gitwara igihe hamwe nibuka bikura hamwe na kare yuburebure (O (n ^ 2)). Kwitonda kumurongo byongeye kubara kuburyo igiciro gikura gusa umurongo (O (n)). Igitekerezo cyingenzi: softmax kwitondera ni softmax (QK ^ T) V, ariko niba usimbuye softmax hamwe nikarita yerekana ikarita phi, ubona phi (Q) (phi (K) ^ T V). Kuberako kugwiza matrix bifitanye isano, ubara phi (K) ^ T V ubanza (mato mato mato mato mato), ukirinda amanota manini n-by-n amanota yose. Abakora, guhera kuri Google muri 2020, bituma ibi bigereranywa kwizerwa rya softmax yukuri ukoresheje FAVOR + (Byihuta Byihutirwa Binyuze muburyo bwiza bwa Orthogonal Random ibiranga), ushushanya ibipimo bitunguranye bituma igereranya ryintangiriro ritabogamye kandi rihamye.

Gushyira mu bikorwa Isi

Gutunganya genomic cyangwa proteine ndende aho kwitabwaho kwadratic byuzuye byananiza ububiko bwa GPU

Urwego-urwego rwincamake hejuru ya raporo ndende cyane nta gucogora, ukoresheje umugozi-wuburyo bwumugongo

Uburyo bwiza-buringaniye bwamajwi cyangwa ibihe-byerekana urugero aho urukurikirane rugera ku bihumbi mirongo

Kugabanya ibiciro byimyanzuro mugihe kirekire-biganiro biganirwaho mugusimbuza ibice bimwe bya softmax hamwe numurongo-wo kwitondera ibintu

Ejo hazaza h'umurongo wo kwitondera no gukora intangiriro

Kwitonda kumurongo ukunze gukurikira softmax kubwiza, bityo umurima ugahuza kuri Hybride: imiterere-yimiterere ya leta (Mamba), umurongo werekeza kumurongo, hamwe nubwubatsi buvanga ibice bike byuzuye-byuzuye hamwe numurongo umwe. Mugihe imiterere ya Windows isunika kuri miriyoni yikimenyetso, umurongo hamwe na sub-quadratic uburyo bugenda bushishikaza kubiciro, kandi uburyo-bwo-buryo-bwo-buryo-bwo-buryo-bwo-buryo-busubirwamo burimo gusubirwamo kugirango habeho gufata neza no kwerekana ibikoresho.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Linear Attention and Performer Kernels quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is Linear Attention and Performer Kernels?

Icyerekezo cyumurongo gisimbuza quadratic softmax kwitondera muri Transformers hamwe namayeri yimibare ipima umurongo hamwe nuburebure bwikurikiranya. Abakora nuburyo bwingenzi bugereranya softmax ukoresheje intungamubiri zidasanzwe, gukora urutonde rurerure cyane kubara.

Ni ukubera iki softmax isanzwe yitabwaho cyane hamwe n'uburebure bukurikirana?

Softmax kwitondera igereranya buri jambo ryibimenyetso, ritanga amanota ya n-by-n, bityo igiciro kiyongera nka O (n ^ 2).

Ni ubuhe buryo bw'imibare butuma umurongo witondera kwirinda n-by-n matrix?

Kuberako kugwiza matrix bifitanye isano, urashobora kubara phi (K) ^ T V ubanza, mato mato mato mato, aho kuba phi (Q) phi (K) ^ T.

Ni ubuhe buryo bwa FAVOR + bukora neza?

FAVOR + ikoresha uburyo bwiza bwa orthogonal idasanzwe kugirango igereranye intungamubiri ya softmax itabanje gukora matrix yuzuye.

Kuki Performer ikoresha ibintu byiza bidasanzwe aho gukoresha trigonometric yambere?

Ibintu byiza biranga intangiriro igereranya ibitari byiza, wirinda ihungabana nagaciro keza byagaragaye mbere yikarita / cos ikarita.

Ni ubuhe buryo bugereranijwe bwa Performer-stil umurongo utondekanya muburyo bukurikiranye n?

Mugutondekanya kubara kandi ntuzigere wubaka matrike ya n-by-n, umunzani wigiciro ugereranije nuburebure bwikurikiranya.