Ntị Linear na kernel ndị na-eme ihe
Nlebara anya Linear na-eji aghụghọ mgbakọ na mwepụ na-eji ogologo usoro dochie anya quadratic softmax na Transformers.
Nchịkọta
Performer is a landmark method that approximates softmax using random feature kernels, making very long sequences computationally affordable.
Ime miri emi
Nleba anya ngbanwe ọkọlọtọ na-agbakọ akara n'etiti akara ngosi abụọ ọ bụla, na-efu oge na ebe nchekwa na-eto n'akụkụ ogologo ogologo (O(n^2)). Nleba anya n'ahịrị na-edegharị mgbakọ ahụ ka ọnụ ahịa na-eto naanị n'ahịrị (O(n)). Echiche bụ isi: nlebara anya softmax bụ softmax (QK ^ T) V, mana ọ bụrụ na iji kernel feature map phi dochie softmax, ị ga-enweta phi (Q) (phi (K) ^T V). N'ihi na ịba ụba matrix bụ ihe mmekọ, ị na-agbakọ phi(K)^T V mbụ (obere d-by-d matrix), na-ezere nnukwu n-by-n akara matrix kpamkpam. Onye na-eme ihe, site na Google na 2020, na-eme ka nke a bụrụ nkwenye kwesịrị ntụkwasị obi nke ezi softmax site na iji FAVOR + (Ntị ngwa ngwa Via nti Orthogonal Random atụmatụ), na-esepụta amụma enweghị usoro nke na-eme atụmatụ kernel enweghị mmasị na kwụsie ike.
Nghọta nka nka
FAVOR+ Performer na-atụ aro softmax kernel exp(q.k) na-eji ezigbo random atụmatụ: ọ na-esepụta ajụjụ na igodo site na ntule Gaussian na-enweghị usoro, na-ekwe nkwa nlebara anya nlebara anya na-abụghị nke na-adịghị mma yana na-ezere ọnụọgụ ọnụọgụ nke ndị nleba anya mbụ. Iji njiri mara orthogonal na-ebelata ọdịiche. N'ụzọ dị oke mkpa, matrix n-by-n adịghị eme ihe mgbe ọ bụla, ya mere ebe nchekwa na-adaba site na quadratic ruo linear, na-enye usoro nke iri puku kwuru iri puku token.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke nlebara anya Linear na kernel ndị na-eme ihe
Nlebara anya n'ahịrị dị ọcha na-esokarị softmax na mma, ya mere ubi ahụ na-agbakọta na ngwakọ: ụdị oghere steeti (Mamba), nlebara anya linear gated, na ihe owuwu nke na-agwakọta ọkwa nlebara anya ole na ole na ọtụtụ ahịrị. Ka windo ndị gbara ya gburugburu na-erute n'ọtụtụ nde tokens, usoro ahịrị na nke dị n'okpuru anọ na-adọrọ adọrọ maka ọnụ ahịa, a na-atụgharịkwa nlebara anya linear ụdị na-emekarị maka ntinye mgbasa ozi nke ọma na ụdị ngwaọrụ.
Mmejuputa n'ezie n'ụwa
Ịhazi ogologo usoro genomic ma ọ bụ protein ebe nlebara anya zuru oke ga-eme ka ebe nchekwa GPU kwụsị
Nchịkọta ọkwa ọkwa akwụkwọ n'ime akụkọ dị ogologo ogologo na-enweghị nbibi, na-eji ọkpụkpụ azụ ụdị onye na-eme ihe.
Nlegharị anya ọdịyo ma ọ bụ usoro oge dị mma nke ọma ebe usoro dị n'usoro dị iri puku kwuru iri puku nzọụkwụ
Ibelata ọnụ ahịa nrịbama n'ụdị nkata ogologo okwu site n'iji ụdị nlebara anya linear dochie ụfọdụ akwa softmax.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Linear Attention and Performer Kernels quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
RWKV Linear Ntị
Ajụjụ a na-ajụkarị
What is Linear Attention and Performer Kernels?
Nlebara anya Linear na-eji aghụghọ mgbakọ na mwepụ na-eji ogologo usoro dochie anya quadratic softmax na Transformers. Onye na-eme ihe bụ usoro ama ama nke na-eme ka softmax dị ka site na iji kernel atụmatụ enweghị usoro, na-eme usoro ogologo oge dị ọnụ ala.
Kedu ihe kpatara nlebara anya softmax ọkọlọtọ na-adachaghị nke ọma na ogologo usoro?
Nleba anya Softmax na-atụnyere ụdị akara ọ bụla, na-emepụta matriks akara n-by-n, yabụ ọnụ ahịa na-eto ka O (n^2).
Kedu ihe akụrụngwa mgbakọ na mwepụ na-ahapụ nlebara anya n'ahịrị zere matriks n-by-n?
N'ihi na ịba ụba nke matrix bụ ihe mmekọ, ị nwere ike gbakọọ phi(K)^T V mbụ, obere d-by-d matrix, kama phi(Q) phi(K)^T.
Kedu ihe usoro FAVOR+ Performer na-adakọ?
FAVOR+ na-eji ezigbo orthogonal random atụmatụ iji tụọ kernel sọksmax exponential na-emepụtaghị matriks nlebara anya zuru oke.
Kedu ihe kpatara onye na-eme ihe ji eji njirimara enweghị nke ọma karịa nke mbụ trigonometric?
Ngosipụta dị mma na-eme atụmatụ kernel na-abụghị nke na-adịghị mma, na-ezere ndakpọ na ụkpụrụ na-adịghị mma nke kparịrị maapụ mmehie mbụ.
Gịnị bụ odidi mgbagwoju anya nke onye na-eme ụdị linear anya n'usoro ogologo n?
Site n'ịhazigharị mgbakọ na mwepụ ma ghara ịrụpụta matriks n-by-n, ọnụ ahịa ọnụ ahịa n'ahịrị na ogologo usoro.