Modèlu attention bu néew
Bayyi xel bu néew dafay tax Transformers gëna xéewale ndax dafay bàyyi token bu nekk mu bàyyi xel ci benn subset buñ tànn bu baax ci yeneen token yi, du leen ñépp.
Résumé
This trades a little global reach for big savings in memory and compute on long sequences.
Plongeur bu xóot
Foofu sa bopp bu mat sëkk dafay méngale bépp token ak bépp token, kon njëg bi dafay màgg ak kaare guddaayu toppalante bi, te loolu dafay metti ci këyitu gudd yi. Li gëna néew luñu koy bàyyi xel mooy wecci motif bu dëgër bi ak motif bu jaxasoo. Njàngat yiñ gëna xam ñooy palanteer buy gliise (bërëb bi), fu token bu nekk di gis dëkkandoo yi ko jege kese; motif strided wala dilated yuy tëb ci kanam ngir yegg ci contexte bu sori ci anam wu yomb; ak jetons global, yenn position yu amul fenn yuy toppatoo lépp te lépp di toppatoo, di liggéey ni hub yiy joxe xibaar. Modèle yu melni Longformer, BigBird, ak Sparse Transformer dañu boole limu lëkkaloo yi di màgg ci anam wu ligneer ci barabu quadratique, loolu mooy tax ñu mëna am ay junni junni wala ay fukki junni token.
Gis-gis xarala
Lu moy matrix attention N-by-N bu mat, sparse attention dafay xayma ay duggal yuñ tann, lu ci bari dafay boole benn palanteer bu dëkk bi ak ay liiñ ak kolon yu néew. BigBird dafa firndeel bu baax ni boole lëkkaloo yu bari, palanteer, ak globaal dafay baña bàyyi xel ci xam-xam bu mat sëkk, boole ci wàññi jafe-jafe yi ci O (N kaare) dem ci O (N). Kernel yu am doole yi dañuy sànni duggal yiñ maskeer yépp, duñu leen di xayma ba noppi di leen nul.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu motif yu néew yi ñuy bàyyi xel
Li gëna néew luñu koy bàyyi xel mooy gëna am solo ci modeling contexte bu yàgg, di gëna boole ak kernel yuñ gëna defar lu melni FlashAttention ak ak sparsity bu jàng wala buy dinaamise buy tànn ban token lañu wara bàyyi xel ci dugal bu nekk. Ginaaw palanteer yi dañuy dem ba ci ay milioŋ ciy jeton, stack ibrid yi dañuy jaxase ay couche yu bari, yu dëgër, ak espace-state. Xaarandi kernel yu bari te xam hardware bi ak bàyyi xel ci routing ngir wéy di wàññi njëgu jàng ay dugal yu gudd lool.
Doxal ci àdduna dëgg
Longformer liggéey ci këyitu science yépp wala këyitu yoon ci benn yoon jëfandikoo palanteer buy gliise boole ci bàyyi xel ci àdduna bi
BigBird dafay jëfandikoo ay tontu ci laaj yu gudd ak genomics yu toppalante ak bàyyi xel ci escale lineaire
Résumé ab mbind bu guddu téere fu bàyyi xel bu mat sëkk di jeexal mémoire GPU
Seet ak sistemu waxtaan yu gudd yuy jëfandikoo ay jeton hub global ngir yónnee ay leeral yu am solo ci ay junni jeton
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sparse Attention Patterns quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Block-Sparse ak Natif Sparse
Laaj yi ñuy faral di laaj
What is Sparse Attention Patterns?
Bayyi xel bu néew dafay tax Transformers gëna xéewale ndax dafay bàyyi token bu nekk mu bàyyi xel ci benn subset buñ tànn bu baax ci yeneen token yi, du leen ñépp. Loolu dafay jënd tuuti yegg ci àdduna bi ngir sakkanal memory bu bari ak xayma ci toppalante yu gudd.
Lan moo waral bàyyi xel ci sa bopp lu seer lool ci toppalante yu gudd?
Foofu yépp dafay méngale token bu nekk ak beneen token bu nekk, loolu mooy jox O (N carré) njëg ci diir ak mémoire.
Luy palanteer buy gliise (local)?
Local attention dafay tënk gis-gis bu token bu nekk ci palanteer bu takku bu token yi ko wër, wàññi njëg yi bu baax.
Lan mooy njariñu 'token global' ci lu néew luñu koy bàyyi xel?
Yenn jeton global yu lëkkaloo ak bépp position, may xibaar bi mu jaar ci sequence bi yépp ci anam wu yomb.
Ban model moo firndeel ni boole bàyyi xel ci anam wu bari, palanteer, ak bàyyi xel ci àdduna bi dafay wéy di fësal xel mi yépp?
BigBird dafa wane ni motif bu bariwul bi ap aproksimatër universel la ci fonction yu toppalante yi ci di scale ci anam wu ligneer.
Ci gàttal, naka lay demee ba limu lëkkaloo yi di yokku ci barab bu ñu defar bu baax?
Suñu tënk lëkkaloo yi ci palanteer yi ci dëkk bi boole ci yenn lëkkalekaay yu mag, lëkkaloo yépp dañuy màgg ci anam wu ligneer.