Ụdị nlebara anya dị obere
Nlebara anya na-adịghị ahụkebe na-eme ka ndị na-agbanwe agbanwe dị ọnụ ala site n'ikwe ka akara ngosi ọ bụla gaa na naanị mpaghara akara ndị ọzọ ejiri nlezianya họrọ karịa ha niile.
Nchịkọta
This trades a little global reach for big savings in memory and compute on long sequences.
Ime miri emi
Nlebara anya onwe onye zuru oke na-atụnyere akara ngosi ọ bụla na akara ngosi ọ bụla ọzọ, ya mere ọnụ ahịa na-eto na square nke ogologo usoro, nke na-egbu mgbu maka akwụkwọ dị ogologo. Nlebara anya na-adịghị ahụkebe na-eji nke ahaziri ahazi dochie ụkpụrụ dị oke egwu. Ihe ndị a na-ahụkarị gụnyere nlebara anya na windo sliding (mpaghara), ebe akara ọ bụla na-ahụ naanị ndị agbata obi dị nso; usoro agbagoro agbagọ ma ọ bụ gbasaa nke na-amali n'ihu iji ruo ebe dị anya n'ọnụ ọnụ ala; na akara ngosi zuru ụwa ọnụ, ọnọdụ ole na ole pụrụ iche na-aga ihe niile na ihe niile na-aga, na-eme dị ka ebe ozi. Ụdị dị ka Longformer, BigBird, na Sparse Transformer na-ejikọta ndị a ka ọnụ ọgụgụ njikọ ahụ na-eto n'ụzọ ziri ezi kama ịnọ n'akụkụ anọ, na-eme ka ọnọdụ puku kwuru puku ruo iri puku kwuru iri puku token.
Nghọta nka nka
Kama nlebara anya N-by-N zuru oke, nlebara anya na-enweghị atụ na-agbakọ naanị ndenye ahọpụtara, na-abụkarị njikọ nke windo mpaghara yana obere ahịrị na kọlụm zuru ụwa ọnụ. BigBird gosipụtara n'ụzọ doro anya na ijikọta random, windo, na njikọ zuru ụwa ọnụ na-echekwa nkwupụta echiche nke nlebara anya zuru oke ma na-ebelata mgbagwoju anya site na O(N squared) n'ebe O(N). Mkpụrụ ndụ dị mma na-amafe ndenye ndị kpuchiri ekpuchi kpamkpam kama ịgbakọ ma mebie ha.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke ụkpụrụ nlebara anya dị obere
Nlebara anya na-adịchaghị mkpa ka bụ ihe dị n'ụdị ihe nlegharị anya ogologo oge, na-ejikọta ya na kernels kachasị dị ka FlashAttention yana nhụsianya mmụta ma ọ bụ ike siri ike nke na-ahọrọ akara ngosi ị ga-aga na ntinye ọ bụla. Ka windo ndị gbara ya gburugburu na-agbatị ruo ọtụtụ nde akara ngosi, nchịkọta ngwakọ na-agwakọta obere oghere, nnukwu oghere na ọkwa steeti. Na-atụ anya kernel ndị na-amata ngwaike na nlebara anya dabere ka ọ nọgide na-ebelata ọnụ ahịa ịgụ ihe ntinye ogologo.
Mmejuputa n'ezie n'ụwa
Ogologo akwụkwọ na-ahazi akwụkwọ sayensị niile ma ọ bụ akwụkwọ iwu n'otu ngafe site na iji sliding-window gbakwunyere nlebara anya zuru ụwa ọnụ
BigBird na-ejizi azịza ajụjụ ogologo akwụkwọ yana usoro genomics nwere nlebara anya n'ahịrị.
Na-achịkọta ederede ogologo akwụkwọ ebe nlebara anya zuru oke ga-agwụ ebe nchekwa GPU
Sistemu nkata eweghachi na ogologo ọnọdụ nke na-eji akara hub zuru ụwa ọnụ na-ebugharị ozi ndị bụ isi gafee puku kwuru puku token.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sparse Attention Patterns quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nleba anya nleba anya nke mgbochi na obere ala
Ajụjụ a na-ajụkarị
What is Sparse Attention Patterns?
Nlebara anya na-adịghị ahụkebe na-eme ka ndị na-agbanwe agbanwe dị ọnụ ala site n'ikwe ka akara ngosi ọ bụla gaa na naanị mpaghara akara ndị ọzọ ejiri nlezianya họrọ karịa ha niile. Nke a na-azụta ntakịrị ohere zuru ụwa ọnụ maka nnukwu ego nchekwa na ebe nchekwa ma gbakọọ na ogologo usoro.
Kedu ihe kpatara nlebara anya onwe onye zuru oke ji dị ọnụ na ogologo usoro?
Nlebara anya zuru oke na-atụnyere akara ọ bụla na akara ngosi ọ bụla, na-enye ọnụ ahịa O(N squared) na oge na ebe nchekwa.
Kedu ihe bụ nlebara anya na windo slide (mpaghara)?
Nlebara anya mpaghara na-egbochi echiche akara ngosi nke ọ bụla na mpio edobere nke akara ngosi gbara ya gburugburu, na-ebelata ọnụ ahịa nke ukwuu.
Gịnị bụ ebumnobi nke 'ihe nleba anya zuru ụwa ọnụ' na nlebara anya nke ọma?
Akara akara ole na ole zuru ụwa ọnụ jikọtara na ọnọdụ niile, na-ahapụ ozi ka ọ gafee usoro niile ọnụ ala.
Kedu ihe nlereanya gosipụtara na ijikọta enweghị usoro, windo, na nlebara anya zuru ụwa ọnụ na-edobe nlebara anya zuru oke?
BigBird gosipụtara ụkpụrụ ya na-adịghị ahụkebe bụ ihe zuru ụwa ọnụ nke ọrụ usoro mgbe ọ na-agbakọ n'ahịrị.
Kedu ka ọnụọgụ njikọ ga-esi gbakọọ na nlebara anya nke emebere nke ọma?
Site na ịmachi njikọ na windo mpaghara gbakwunyere njikọ zuru ụwa ọnụ ole na ole, njikọ zuru ezu na-eto eto n'ahịrị.