Ntibisanzwe
Kwitonda gake bituma Transformers ihendutse kureka buri kimenyetso cyitabira gusa igice cyatoranijwe neza cyibindi bimenyetso aho kuba byose.
Incamake
This trades a little global reach for big savings in memory and compute on long sequences.
Kwibira cyane
Kwiyitaho kwuzuye kugereranya buri kimenyetso nikindi kimenyetso, bityo igiciro gikura hamwe na kare yuburebure bwikurikiranya, bikababaza inyandiko ndende. Ubwitonzi budasanzwe busimbuza ishusho yuzuye nuburyo bwubatswe. Ibishushanyo bisanzwe birimo kunyerera-idirishya (hafi) kwitondera, aho buri kimenyetso kibona abaturanyi gusa; Inzira zigenda cyangwa zagutse zisimbuka imbere kugirango zigere ku ntera ya kure ihendutse; n'ibimenyetso byisi yose, imyanya mike idasanzwe yitabira ibintu byose kandi ko ibintu byose byitabira, bikora nkamakuru yamakuru. Moderi nka Longformer, BigBird, na Sparse Transformer ihuza ibi bityo umubare rusange wibihuza ukura hafi kumurongo aho kuba inshuro enye, bigafasha ibice byibihumbi kugeza ku bihumbi mirongo.
Ubushishozi
Aho kugirango N-by-N byuzuye matrike, kwitonda gake kubara ibyatoranijwe gusa, akenshi ihuriro ryidirishya ryaho hamwe nintoki zumurongo wisi hamwe ninkingi. BigBird izwi cyane ko guhuza ibintu bidasanzwe, idirishya, hamwe nisi yose bihuza ibitekerezo byerekana ibitekerezo byuzuye mugihe bigabanya kuva kuri O (N kwaduka) yerekeza kuri O (N). Intungamubiri zikora neza zisimbuka ibyanditswemo byuzuye aho kubara noneho zeru.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza Kudakunze kwitabwaho
Kwitonda gake bikomeza kuba ingenzi kumurongo muremure-wo kwerekana imiterere, bigenda bihuzwa hamwe nintungamubiri zitezimbere nka FlashAttention hamwe nubumenyi bwize cyangwa imbaraga zidasanzwe zitoranya ibimenyetso byo kwitabira kwinjiza. Mugihe imiterere ya Windows irambuye miriyoni yikimenyetso, ibice bivangavanze bivanga gake, byuzuye, hamwe na leta-umwanya. Tegereza ibyuma-bimenyesha ibyuma bito hamwe na maritire ishingiye kubitekerezo kugirango ukomeze kugabanya ikiguzi cyo gusoma inyongeramusaruro ndende.
Gushyira mu bikorwa Isi
Longformer itunganya impapuro zose zubumenyi cyangwa inyandiko zemewe mumurongo umwe ukoresheje kunyerera-idirishya wongeyeho isi yose
BigBird ikora inyandiko-ndende ikibazo gisubiza hamwe na genomics ikurikiranye hamwe n'umurongo-wo gupima
Incamake yibitabo-uburebure bwigitabo aho kwitondera byuzuye byananiza GPU yibuka
Gushakisha hamwe na sisitemu ndende yo kuganira ikoresha isi ya hub tokens kugirango yerekane amakuru yingenzi kubihumbi
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sparse Attention Patterns quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Guhagarika-Ntibisanzwe na Kavukire Ntibisanzwe
Ibibazo bikunze kubazwa
What is Sparse Attention Patterns?
Kwitonda gake bituma Transformers ihendutse kureka buri kimenyetso cyitabira gusa igice cyatoranijwe neza cyibindi bimenyetso aho kuba byose. Ibi bigurisha gato kwisi yose kubitsa binini murwibutso no kubara kumurongo muremure.
Kuki kwiyitaho byuzuye bihenze kurikurikirane?
Ubwitonzi bwuzuye bugereranya buri kimenyetso nikindi kimenyetso, gitanga O (N squared) igiciro mugihe no kwibuka.
Niki kunyerera-idirishya (hafi) kwitondera?
Ibitekerezo byaho bigabanya buri kimenyetso cyo kureba kumadirishya ihamye yikimenyetso, kugabanya ibiciro kuburyo bugaragara.
Niyihe ntego ya 'global tokens' mubitekerezo bidakunze kubaho?
Ibimenyetso bike kwisi bihuza imyanya yose, kureka amakuru atembera murwego rwose ruhendutse.
Ni ubuhe buryo bwagaragaje ko guhuza ibintu bidasanzwe, idirishya, hamwe nisi yose bikomeza kwitabwaho byuzuye?
BigBird yerekanye imiterere yayo idasanzwe ni igereranya rusange ryimikorere ikurikirana mugihe igereranije neza.
Hafi yukuntu umubare wibihuza wapima muburyo bwateguwe neza?
Mugabanye guhuza Windows yaho wongeyeho amahuza make yisi yose, amahuza yose akura kumurongo.