Tsare-tsare Hankali
Hankalin da ya rage yana sa masu canza canjin su zama mai rahusa ta hanyar barin kowace alamar ta halarci wani yanki da aka zaɓa kawai na wasu alamu maimakon duka.
Dubawa
This trades a little global reach for big savings in memory and compute on long sequences.
Zurfafa nutsewa
Cikakken hankalin kai yana kwatanta kowane alamar da kowane alamar, don haka farashi yana girma tare da murabba'in tsayin jeri, wanda ya zama mai raɗaɗi ga dogon takardu. Hankali mara kyau yana maye gurbin tsari mai yawa tare da tsari mai tsari. Zane-zane na gama-gari sun haɗa da kulawar taga mai zamewa (na gida), inda kowace alama ke gani kawai maƙwabta na kusa; m ko faffadan alamu waɗanda ke tsallake gaba don isa ga mahallin mai nisa da rahusa; da alamomin duniya, ƴan matsayi na musamman waɗanda ke halartar kowane abu kuma duk abin da ke halarta, suna aiki azaman cibiyoyin bayanai. Model kamar Longformer, BigBird, da Sparse Transformer sun haɗu da waɗannan don haka jimillar haɗin haɗin gwiwa yana girma kusan layi ɗaya maimakon quadratically, yana ba da damar mahallin dubbai zuwa dubun dubatar alamun.
Fahimtar Fasaha
Maimakon cikakken matrix N-by-N, hankali mara hankali yana ƙididdige shigarwar da aka zaɓa kawai, sau da yawa ƙungiyar tagar gida da ɗimbin layuka da ginshiƙai na duniya. BigBird sanannen ya tabbatar da cewa haɗa bazuwar, taga, da haɗin haɗin gwiwar duniya yana kiyaye ka'idar bayanin cikakken hankali yayin da rage sarƙaƙƙiya daga O(N squared) zuwa O(N). Ingantattun kernels suna tsallake shigarwar da aka rufe su gaba ɗaya maimakon yin lissafi sannan a cire su.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Tsararrun Hankali
Hankalin da ya rage ya kasance tsakiya ga ƙirar yanayi mai tsayi, yana ƙara haɗe tare da ingantattun kernels kamar FlashAttention kuma tare da koyo ko tsauri mai ƙarfi wanda ke ɗaukar alamun da za a halarta ga kowace shigarwa. Kamar yadda mahallin windows ke shimfiɗawa zuwa miliyoyin alamu, ɗimbin ɗimbin ɗimbin ɗimbin ɗimbin ɗimbin ɗimbin yawa suna haɗo ɗimbin yadudduka masu yawa, masu yawa, da yadudduka na sararin samaniya. Yi tsammanin kernels masu saurin sanin kayan masarufi da kulawa mai tushe don ci gaba da raguwar farashin karanta bayanai masu tsayi sosai.
Aiwatar da Gaskiyar Duniya
Mai daɗe yana sarrafa duk takaddun kimiyya ko takaddun doka a cikin fasfo ɗaya ta amfani da taga mai zamewa tare da kulawar duniya
BigBird yana kula da dogon-takardun tambaya amsa da jeri na genomics tare da kulawar linzamin kwamfuta
Taƙaita rubutun tsawon littafi inda cikakken hankali zai ƙare ƙwaƙwalwar GPU
Maidowa da tsarin tattaunawa na dogon lokaci waɗanda ke amfani da alamun cibi na duniya don sarrafa mahimman bayanai a cikin dubunnan alamu.
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sparse Attention Patterns quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Toshe-Sparse da Hankali na ɗan ƙasa
Tambayoyin da ake yawan yi
What is Sparse Attention Patterns?
Hankalin da ya rage yana sa masu canza canjin su zama mai rahusa ta hanyar barin kowace alamar ta halarci wani yanki da aka zaɓa kawai na wasu alamu maimakon duka. Wannan yana cinikin ɗan isar duniya don babban tanadi a ƙwaƙwalwar ajiya da ƙididdige jerin dogayen jeri.
Me yasa cikakken hankalin kai yayi tsada akan dogon jerin?
Cikakken hankali yana kwatanta kowane alamar zuwa kowane alamar, yana ba da O(N squared) farashi a cikin lokaci da ƙwaƙwalwar ajiya.
Menene kulawar taga mai zamewa (na gida)?
Hankalin gida yana takurawa kowane alamar alama zuwa ƙayyadaddun taga alamun kewaye, rage tsada sosai.
Menene maƙasudin 'alamu na duniya' a cikin rashin kulawa?
ƴan alamun duniya suna haɗawa zuwa kowane matsayi, suna barin bayanai su gudana cikin jerin duka cikin arha.
Wanne samfuri ne ya tabbatar da cewa haɗa bazuwar, taga, da hankalin duniya yana kiyaye cikakkiyar kulawa?
BigBird ya nuna ƙarancin tsarin sa shine ma'auni na duniya na ayyukan jeri yayin da yake yin ƙima a layi.
Kusan ta yaya yawan haɗin haɗin ke yin ma'auni a cikin ingantaccen tsari da hankali?
Ta iyakance haɗi zuwa windows na gida da ƴan hanyoyin haɗin duniya, jimlar haɗin gwiwa suna girma akan layi.