HAGAHA Farsamada
Feejignaan toosan iyo Kernels hawl-wadeen
Dareenka tooska ah wuxuu ku beddelaa dareenka softmax quadratic ee Transformers oo leh khiyaamo xisaabeed oo si toos ah u cabbiraysa dhererka isku xigxiga.
Dhageyso hagahanAI-waa laga sheekeeyay · ~2 min
Boggaan2 daqiiqo akhri
Dulmar
Performer waa hab calaamad ah oo ku qiyaasa softmax iyadoo la adeegsanayo kernels sifada random, taasoo ka dhigaysa taxane aad u dheer oo xisaabin ah oo la awoodi karo.
quusid qoto dheer
Dareenka beddelka caadiga ah wuxuu xisaabiyaa dhibcaha u dhexeeya lammaane kasta, kharashka wakhtiga iyo xusuusta ku koraan labajibbaaran dhererka isku xigxiga (O(n^2)). Dareenka tooska ah ayaa dib u qoraa xisaabinta sidaas darteed qiimuhu wuxuu u koraa kaliya toosan (O(n)). Fikradda muhiimka ah: dareenka softmax waa softmax (QK^T) V, laakiin haddii aad ku bedesho softmax khariidadda kernel feature phi, waxaad helaysaa phi (Q) (phi (K) ^T V). Sababtoo ah isku dhufashada matrixku waa wadaag, waxaad xisaabinaysaa phi(K)^T V marka hore (matrix d-by-d yar), adigoo iska ilaalinaya shaxanka dhibcaha n-by-n gabi ahaanba. Sameeyaha, laga bilaabo Google sanadka 2020, ayaa tan ka dhigaya qiyaas aamin ah oo softmax run ah iyadoo la adeegsanayo FAVOR+ (Fast Attention Via togan Orthogonal Random features), isagoo sawiraya saadaasha random ee ilaalinaya qiyaasaha kernelku mid aan eex lahayn oo xasiloon.
Dhaqangelinta Adduunka-dhabta ah
Ka baaraandegidda taxanaha genomic ama borotiinka dheer halkaas oo dareenka quadratic-ka buuxa uu daali doono xusuusta GPU-da
Soo koobida heerka-dokumentiga ee warbixinada aadka u dheer iyadoon la kala goyn, iyadoo la adeegsanayo laf dhabarta qaab-fuliyaha
Hab-samaynta maqal-dheer ee wax-ku-oolka ah ama qaab-waqtiga-xilliyeedka oo taxanuhu uu gaadho tobanaan kun oo tallaabo
Hoos u dhigida qiimaha go'aaminta moodooyinka wada sheekaysiga muddada-dheer iyadoo lagu beddelayo lakabyo softmax ah noocyo kala duwanaansho-toos ah
Mustaqbalka Feejignaanta Tooska ah iyo Kernels hawl-wadeenada
Dareenka toosan ee saafiga ah wuxuu inta badan raadiyaa softmax tayada, sidaas darteed garoonku wuxuu ku soo ururayaa isku-dhafka: moodooyinka gobolka-meel (Mamba), dareenka toosan ee la daboolay, iyo qaab-dhismeedyo isku dhafan dhowr lakab oo taxadar leh oo leh kuwo badan oo toosan. Sida daaqadaha macnaha guud ay u riixayaan malaayiin calaamado ah, hababka toosan iyo kuwa afar-geesoodka ah ayaa si isa soo taraysa u soo jiidanaya kharashka, iyo dareenka toosan ee qaabka soo noqnoqda ayaa dib loogu eegayaa soo-jeedinta hufan iyo moodooyinka qalabka.
Ku dhaqangelinta
Maxay muhiim u tahay
- Qiimaha iyo miisaaniyada. Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
- Go'aamo cad. Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
- Xakamaynta tayada. Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
Iska ilaali
- Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
- Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
- Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Tallaabooyinka xiga
- Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
- Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
- La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
- U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Warqad bilaash ah
Hel warbixinta maalinlaha ah ee AI
Saddex sheeko oo la xaqiijiyay oo AI ah maalin kasta oo shaqo ah, oo lagu qoray Ingiriis fudud. Bilaash had iyo jeer, aan xayaysiis lahayn.
Hal iimayl maalin kasta oo shaqo ah. Ka bax hal gujin. Marna ma iibinno ama ma wadaagno ciwaankaaga.
Is tijaabi
Take the Linear Attention and Performer Kernels quiz
Fikrad degdeg ah oo ku saabsan jawaab kasta, iyo shahaado la wadaajin karo oo leh aqoonsi la xaqiijin karo markaad ka gudubto koorso.
Taageer waxbarashada AI bilaashka ah. AI Understanding waa 501(c)(3) aan faa'iido doon ahayn — xayaysiis la'aan, albaab lacag ah ma jiro, had iyo jeer. Samee deeq
Su'aalaha soo noqnoqda
Waa maxay Feejignaanta Tooska ah iyo Kernels Performer?
Dareenka tooska ah wuxuu ku beddelaa dareenka softmax quadratic ee Transformers oo leh khiyaamo xisaabeed oo si toos ah u cabbiraysa dhererka isku xigxiga. Performer waa hab calaamad ah oo ku qiyaasa softmax iyadoo la adeegsanayo kernels sifada random, taasoo ka dhigaysa taxane aad u dheer oo xisaabin ah oo la awoodi karo.
Waa maxay sababta feejignaanta softmax-ga caadiga ahi u liidato dhererka isku xigxiga?
Dareenka Softmax wuxuu isbarbar dhigayaa lammaane kasta, isagoo soo saaraya matrix dhibcaha n-by-n, markaa kharashku wuxuu u koraa sida O (n^2).
Waa maxay hantida xisaabeed ee u oggolaanaysa dareenka toosan inay ka fogaato matrixka n-by-n?
Sababtoo ah isku dhufashada matrixku waa wada shaqayn, waxaad xisaabin kartaa phi(K)^T V marka hore, matrix d-by-d yar, halkii aad ka ahaan lahayd phi(Q) phi(K)^T.
Waa maxay habka FAVOR+ ee Performer?
FAVOR+ waxay isticmaashaa astaamo random orthogonal togan si ay u qiyaasto jibbaarada softmax kernel iyada oo aan la samayn matrix dareenka buuxa.
Muxuu Performer u adeegsadaa sifooyin random togan halkii uu isticmaali lahaa kuwii hore ee trigonometric?
Tilmaamaha togan waxay ku hayaan qiyaasaha kernel-ku kuwo aan xumaan lahayn, iyaga oo iska ilaalinaya xasilloonida iyo qiyamka taban ee aafeeyay dembigii hore ee khariidadaha.
Waa maxay kakanaanta qiyaasta ee dareenka toosan ee qaab-fuliyaha ee dhererka isku xigxiga n?
Adiga oo dib u habaynaya xisaabinta oo aan waligiis dhisin shaxanka n-by-n, qiimayaashu waxay si toos ah u cabbirayaan dhererka isku xigxiga.
Sii wad waxbarashada
Tilmaamaha la xidhiidha
Tilmaamayaal badan ayaa loo doortay mawduucan