Feejignaan toosan iyo Kernels hawl-wadeen
Dareenka tooska ah wuxuu ku beddelaa dareenka softmax quadratic ee Transformers oo leh khiyaamo xisaabeed oo si toos ah u cabbiraysa dhererka isku xigxiga.
Dulmar
Performer waa hab calaamad ah oo ku qiyaasa softmax iyadoo la adeegsanayo kernels sifada random, taasoo ka dhigaysa taxane aad u dheer oo xisaabin ah oo la awoodi karo.
quusid qoto dheer
Dareenka beddelka caadiga ah wuxuu xisaabiyaa dhibcaha u dhexeeya lammaane kasta, kharashka wakhtiga iyo xusuusta ku koraan labajibbaaran dhererka isku xigxiga (O(n^2)). Dareenka tooska ah ayaa dib u qoraa xisaabinta sidaas darteed qiimuhu wuxuu u koraa kaliya toosan (O(n)). Fikradda muhiimka ah: dareenka softmax waa softmax (QK^T) V, laakiin haddii aad ku bedesho softmax khariidadda kernel feature phi, waxaad helaysaa phi (Q) (phi (K) ^T V). Sababtoo ah isku dhufashada matrixku waa wadaag, waxaad xisaabinaysaa phi(K)^T V marka hore (matrix d-by-d yar), adigoo iska ilaalinaya shaxanka dhibcaha n-by-n gabi ahaanba. Sameeyaha, laga bilaabo Google sanadka 2020, ayaa tan ka dhigaya qiyaas aamin ah oo softmax run ah iyadoo la adeegsanayo FAVOR+ (Fast Attention Via togan Orthogonal Random features), isagoo sawiraya saadaasha random ee ilaalinaya qiyaasaha kernelku mid aan eex lahayn oo xasiloon.
Aragtida Farsamada
FAVOR ee hawlwadeenka Isticmaalka sifada aan sugnayn ee orthogonal waxay yaraynaysaa kala duwanaanshaha. Muhiimad ahaan, matrixka dareenka n-by-n waligiis lama hirgelin, sidaa darteed xasuustu waxay hoos uga dhacdaa quadratic ilaa toosan, taasoo awood u siinaysa taxane tobanaan kun oo calaamado ah.
Saamaynta Istiraatijiyadeed
Qiimaha iyo miisaaniyada
Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
Go'aamo cad
Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
Xakamaynta tayada
Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
Mustaqbalka Feejignaanta Tooska ah iyo Kernels hawl-wadeenada
Dareenka toosan ee saafiga ah wuxuu inta badan raadiyaa softmax tayada, sidaas darteed garoonku wuxuu ku soo ururayaa isku-dhafka: moodooyinka gobolka-meel (Mamba), dareenka toosan ee la daboolay, iyo qaab-dhismeedyo isku dhafan dhowr lakab oo taxadar leh oo leh kuwo badan oo toosan. Sida daaqadaha macnaha guud ay u riixayaan malaayiin calaamado ah, hababka toosan iyo kuwa afar-geesoodka ah ayaa si isa soo taraysa u soo jiidanaya kharashka, iyo dareenka toosan ee qaabka soo noqnoqda ayaa dib loogu eegayaa soo-jeedinta hufan iyo moodooyinka qalabka.
Dhaqangelinta Adduunka-dhabta ah
Ka baaraandegidda taxanaha genomic ama borotiinka dheer halkaas oo dareenka quadratic-ka buuxa uu daali doono xusuusta GPU-da
Soo koobida heerka-dokumentiga ee warbixinada aadka u dheer iyadoon la kala goyn, iyadoo la adeegsanayo laf dhabarta qaab-fuliyaha
Hab-samaynta maqal-dheer ee wax-ku-oolka ah ama qaab-waqtiga-xilliyeedka oo taxanuhu uu gaadho tobanaan kun oo tallaabo
Hoos u dhigida qiimaha go'aaminta moodooyinka wada sheekaysiga muddada-dheer iyadoo lagu beddelayo lakabyo softmax ah noocyo kala duwanaansho-toos ah
Khatarta & Dariiqyada Ilaalada
Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Qorshe Hawleedka Dhaqangelinta
Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Linear Attention and Performer Kernels quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
RWKV Feejignaan toosan
Su'aalaha soo noqnoqda
Waa maxay Feejignaanta Tooska ah iyo Kernels Performer?
Dareenka tooska ah wuxuu ku beddelaa dareenka softmax quadratic ee Transformers oo leh khiyaamo xisaabeed oo si toos ah u cabbiraysa dhererka isku xigxiga. Performer waa hab calaamad ah oo ku qiyaasa softmax iyadoo la adeegsanayo kernels sifada random, taasoo ka dhigaysa taxane aad u dheer oo xisaabin ah oo la awoodi karo.
Waa maxay sababta feejignaanta softmax-ga caadiga ahi u liidato dhererka isku xigxiga?
Dareenka Softmax wuxuu isbarbar dhigayaa lammaane kasta, isagoo soo saaraya matrix dhibcaha n-by-n, markaa kharashku wuxuu u koraa sida O (n^2).
Waa maxay hantida xisaabeed ee u oggolaanaysa dareenka toosan inay ka fogaato matrixka n-by-n?
Sababtoo ah isku dhufashada matrixku waa wada shaqayn, waxaad xisaabin kartaa phi(K)^T V marka hore, matrix d-by-d yar, halkii aad ka ahaan lahayd phi(Q) phi(K)^T.
Waa maxay habka FAVOR+ ee Performer?
FAVOR+ waxay isticmaashaa astaamo random orthogonal togan si ay u qiyaasto jibbaarada softmax kernel iyada oo aan la samayn matrix dareenka buuxa.
Muxuu Performer u adeegsadaa sifooyin random togan halkii uu isticmaali lahaa kuwii hore ee trigonometric?
Tilmaamaha togan waxay ku hayaan qiyaasaha kernel-ku kuwo aan xumaan lahayn, iyaga oo iska ilaalinaya xasilloonida iyo qiyamka taban ee aafeeyay dembigii hore ee khariidadaha.
Waa maxay kakanaanta qiyaasta ee dareenka toosan ee qaab-fuliyaha ee dhererka isku xigxiga n?
Adiga oo dib u habaynaya xisaabinta oo aan waligiis dhisin shaxanka n-by-n, qiimayaashu waxay si toos ah u cabbirayaan dhererka isku xigxiga.