Hankalin Layi na Layi da Ƙwayoyin Ƙwararru
Hankalin layi yana maye gurbin hankali mai laushi mai laushi a cikin masu canzawa tare da dabarar lissafi wanda ke daidaita layi tare da tsayin jeri.
Dubawa
Performer is a landmark method that approximates softmax using random feature kernels, making very long sequences computationally affordable.
Zurfafa nutsewa
Daidaitaccen Mai Canjawa Hankali yana ƙididdige maki tsakanin kowane nau'i biyu na alamu, lokacin kashe kuɗi da ƙwaƙwalwar ajiya waɗanda ke girma tare da murabba'in tsayin jeri (O(n^2)). Hankalin layi yana sake rubuta lissafin don haka farashi yana girma a layi (O(n)) kawai. Mahimmin ra'ayi: hankalin softmax shine softmax (QK^T) V, amma idan kun maye gurbin softmax tare da taswirar kernel phi, zaku sami phi (Q) (phi (K) ^T V). Saboda yawan matrix ɗin haɗin gwiwa ne, kuna ƙididdige phi(K)^T V farko (ƙaramin d-by-d matrix), guje wa babban matrix n-by-n gabaɗaya. Mai yin, daga Google a cikin 2020, ya sanya wannan amintacce kimar softmax na gaskiya ta amfani da FAVOR+ (Saurin Hankali ta hanyar ingantattun siffofi na Orthogonal Random), zana tsinkaya bazuwar da ke kiyaye kimar kwaya ba ta nuna son kai ba.
Fahimtar Fasaha
FAVOR+ na mai yin yana ƙayyadad da softmax kernel exp(q.k) ta amfani da ingantattun siffofi na bazuwar: yana tsara taswirori da maɓalli ta hanyar tsinkayar Gaussian bazuwar da aka nannade cikin ma'auni, yana ba da tabbacin ma'aunin kulawa mara kyau da kuma guje wa rashin daidaituwar lambobi na masu kimantawa a baya. Amfani da sifofin bazuwar orthogonal yana rage bambance-bambance. Mahimmanci, matrix na n-by-n ba a taɓa yin abu ba, don haka ƙwaƙwalwar ajiya tana faɗuwa daga ma'auni zuwa madaidaiciya, yana ba da damar jerin dubun dubatar alamu.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Hankalin Kai tsaye da Kernels masu yin aiki
Hankalin madaidaiciya mai tsafta sau da yawa yana bin softmax akan inganci, don haka filin yana haɗuwa akan hybrids: ƙirar sararin samaniya (Mamba), kulawar madaidaiciyar hankali, da gine-ginen gine-gine waɗanda ke haɗa ƴan yadudduka masu cikakken hankali tare da masu layi ɗaya da yawa. Kamar yadda mahallin windows ke matsawa ga miliyoyin alamu, hanyoyin layin layi da na ƙasa-da-hudu suna ƙara sha'awar farashi, kuma ana sake duba hankalin madaidaiciyar salo na yau da kullun don ingantaccen ra'ayi na yawo da samfuran kan na'urori.
Aiwatar da Gaskiyar Duniya
Sarrafa dogayen tsarin kwayoyin halitta ko furotin inda cikakken kulawar kwatanci zai ƙare ƙwaƙwalwar GPU
Takaitacciyar matakin daftarin aiki akan dogon rahotanni ba tare da gunaguni ba, ta amfani da kashin baya mai salo.
Ingantacciyar tsarin sauti mai tsayi ko ƙirar lokaci-lokaci inda jeri ya wuce dubunnan matakai
Rage ƙimar ƙima a cikin tsarin tattaunawa na dogon lokaci ta hanyar maye gurbin wasu yadudduka na softmax tare da bambance-bambancen hankali
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Linear Attention and Performer Kernels quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
RWKV Hankalin Litattafai
Tambayoyin da ake yawan yi
What is Linear Attention and Performer Kernels?
Hankalin layi yana maye gurbin hankali mai laushi mai laushi a cikin masu canzawa tare da dabarar lissafi wanda ke daidaita layi tare da tsayin jeri. Mai aiwatarwa hanya ce mai alamar ƙasa wacce ke kusantar softmax ta amfani da kernels bazuwar, yin jerin dogayen jeri cikin araha.
Me yasa daidaitaccen hankali softmax ya yi rashin ƙarfi tare da tsayin jeri?
Hankalin Softmax yana kwatanta kowane nau'i na alamu, yana samar da matrix na n-by-n, don haka farashi yana girma kamar O (n^2).
Wace dukiya ta lissafi ce ke ba da hankali ga madaidaiciyar hankali don guje wa matrix n-by-n?
Saboda yawan matrix haɗin gwiwa ne, zaku iya lissafta phi(K)^T V farko, ƙaramin d-by-d matrix, maimakon phi(Q) phi(K)^T.
Menene tsarin FAVOR+ na Performer?
FAVOR+ yana amfani da ingantattun fasalulluka na bazuwar kothogonal don kimanta ma'aunin softmax kernel ba tare da samar da cikakken matrix na hankali ba.
Me yasa Mai yin wasan kwaikwayo ke amfani da ingantattun siffofi bazuwar maimakon na farkon trigonometric?
Kyawawan fasalulluka suna kiyaye ƙididdiga na kernel ba mara kyau ba, guje wa rashin kwanciyar hankali da munanan dabi'u waɗanda suka addabi taswira na farko na zunubi/cos.
Menene madaidaicin madaidaicin kulawar salon mai aiwatarwa a cikin jerin tsayin n?
Ta hanyar sake yin odar lissafi kuma ba za a taɓa gina matrix n-by-n ba, ma'aunin farashi yana daidaita daidai da tsayin jeri.