Qaababka Fiiro Gaarka Leh
Fiiro gaar ah ayaa ka dhigaysa Transformers ka raqiisan iyadoo u oggolaanaysa calaamad kasta inay ka qaybgasho kaliya si taxadar leh loo doortay oo calaamado kale ah halkii ay dhammaantood ahaan lahaayeen.
Dulmar
This trades a little global reach for big savings in memory and compute on long sequences.
quusid qoto dheer
Feejignaanta buuxda waxay is barbardhigtaa calaamad kasta iyo calaamad kasta oo kale, markaa kharashku wuxuu ku koraa labajibbaaran dhererka isku xigxiga, kaas oo xanuun ku noqda dukumentiyada dheer. Fiiro gaar ah ayaa ku beddeleysa qaabka cufan oo leh mid habaysan. Nashqadaha caadiga ah waxaa ka mid ah dareenka simbiriirixidda-daaqadda (maxali ah), halkaas oo calaamad kasta ay aragto kaliya deriska u dhow; habab luntay ama fidsan oo hore u booda si ay u gaadhaan macnaha fog si raqiis ah; iyo calaamado caalami ah, dhowr jagooyin gaar ah oo wax walba ka qayb gala iyo in wax walba ay ka qaybgalaan, u dhaqma sidii xarumaha macluumaadka. Moodooyinka sida Longformer, BigBird, iyo Transformer-ka Sparse waxay isku daraan kuwan si tirada guud ee isku xirnaanta ay u koraan qiyaas ahaan si toos ah halkii ay afar ahaan ka ahaan lahaayeen, taas oo awood u siineysa kumanaan ilaa tobanaan kun oo calaamado ah.
Aragtida Farsamada
Halkii aad ka heli lahayd matrix fiiro gaar ah oo N-by-N ah, fiiro gaar ah ayaa xisaabiya gelida la doortay oo keliya, inta badan ururka daaqadda maxalliga ah iyo sacab muggiis oo saf iyo tiirar caalami ah. BigBird waxay si caan ah u caddeeyeen in isku darka random, daaqada, iyo isku xirka caalamiga ah ay ilaalinayso muujinta aragtiyeed ee dareenka buuxa iyada oo la yareynayo kakanaanta O(N) ee O(N). Kernel-ka wax-ku-oolka ah ayaa ka booda gelitaanka waji-xidhan gabi ahaanba intii ay xisaabin lahaayeen ka dibna eber ka dhigi lahaayeen.
Saamaynta Istiraatijiyadeed
Xawaaraha iyo miisaanka
Socodka shaqada luqaddu si dhakhso leh ayay u socon kartaa iyada oo aan la hurayn joogteynta.
Helitaanka iyo gaarsiinta
Waxay balaadhisaa gelitaanka luqadaha iyo qaababka isgaarsiinta.
Go'aamo cad
Kooxuhu waxay waqti badan ku qaadan karaan xukunka halka otomaatiggu uu qabanayo ku celcelinta.
Mustaqbalka Hababka Fiiro Gaarka ah
Dareenka yar ayaa udub dhexaad u ah qaabaynta mawduuca dheer, oo si sii kordheysa loogu lammaan yahay kernels la hagaajiyay sida FlashAttention iyo labarashada la bartay ama firfircoon ee dooranaysa calaamadihii lagu dhigan lahaa hal-abuurkiiba. Marka daaqadaha macnaha guud ay u fidiyaan malaayiin calaamado ah, xidhmooyinka isku-dhafan waxay isku daraan lakabyo aan badnayn, cufan, iyo heer-goboleed. Filo qalab-ka-og-ogaanshaha kernels-yar-yar iyo feejignaanta ku-saleysan hab-socodka si ay u sii yaraanayso qiimaha wax-akhriska aadka u dheer.
Dhaqangelinta Adduunka-dhabta ah
Ka baaraandegidda dhammaan waraaqaha sayniska ama dukumentiyada sharciga ah ee hal baas iyadoo la adeegsanayo daaqad simbiriirixan iyo dareenka caalamiga ah
BigBird wax ka qabashada su'aalaha dukumeentiga dheer ka jawaabista iyo taxanaha genomics oo leh feejignaan toosan
Soo koobida qoraalka dhererka buugga halkaasoo dareenka buuxa uu baabi'in doono xusuusta GPU
Soo celinta iyo nidaamyada wada sheekaysiga muddada-dheer ee adeegsada calaamadaha xuddunta caalamiga ah si ay u maraan macluumaadka muhiimka ah kumanaan calaamadood
Khatarta & Dariiqyada Ilaalada
Xaqiiqooyinka dhalanteed waxay si deggan u geli karaan warbixinnada, taageerada socodka, ama natiijooyinka cilmi-baarista.
Dareenka degdega ahi wuxuu abuuri karaa natiijooyin aan iswaafaqayn codsiyada la midka ah.
Xogta qoraalka xasaasiga ah ayaa laga yaabaa in la kashifo haddii kontaroolada gelitaanka ay daciif yihiin.
Qorshe Hawleedka Dhaqangelinta
Qeex qaabka wax soo saarka, codka, iyo heerarka tayada ka hor inta aan la baahin.
Jawaabaha salka ku haya ilo lagu kalsoon yahay mar kasta oo saxnidu ay muhiim tahay.
Hayso isbaarada dib u eegista bini aadamka ee wax soo saarka sare.
Lasoco qaababka guuldarada oo dib u leyli dardargelinta ama socodka shaqada si joogto ah.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sparse Attention Patterns quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
Block-Sparse iyo Feejignaan yar oo Dhalad ah
Su'aalaha soo noqnoqda
What is Sparse Attention Patterns?
Fiiro gaar ah ayaa ka dhigaysa Transformers ka raqiisan iyadoo u oggolaanaysa calaamad kasta inay ka qaybgasho kaliya si taxadar leh loo doortay oo calaamado kale ah halkii ay dhammaantood ahaan lahaayeen. Tani waxay ka baayacmushtareysaa in yar oo caalami ah oo la gaaro kaydka weyn ee xusuusta iyo xisaabinta taxanaha dheer.
Waa maxay sababta feejignaanta buuxda ugu qaalisan tahay taxanaha dheer?
Dareenka buuxa wuxuu isbarbar dhigayaa calaamad kasta iyo calaamad kasta, iyadoo siinaya O(N labajibbaaran) kharashka waqtiga iyo xusuusta.
Waa maxay dareenka simbiriirixidda-daaqadda (maxalli)?
Dareenka maxalliga ahi wuxuu xaddidaa aragtida calaamad kasta daaqad go'an oo calaamado ah oo ku xeeran, taasoo dhimaysa qiimaha si weyn.
Waa maxay ujeedada 'calaamadaha caalamiga' ee dareenka aan badnayn?
Calaamado caalami ah oo dhawr ah ayaa ku xidha jagooyinka oo dhan, taas oo u oggolaanaysa in akhbaartu si raqiis ah u socoto dhammaan taxanaha.
Qaabkee caddeeyey in isku darka random, daaqada, iyo dareenka caalamiga ahi ay ilaalinayso muujinta feejignaan buuxda?
BigBird waxay muujisay qaabkeeda yar yar inuu yahay qiyaasaha caalamiga ah ee hawlaha isku xigxiga halka ay u miisaamayso qiyaas ahaan toosan.
Qiyaas ahaan sidee ayay tirada isku xidhka u cabbiraysaa dareenka yar ee sida wanaagsan loo qaabeeyey?
Adigoo xaddidaya isku xirka daaqadaha maxalliga ah iyo dhowr xiriiriye oo caalami ah, wadarta xiriiryadu waxay u koraan si toos ah.