Isku-xidhka Booska Wareega
Rotary Position Embeddings (Rotary Position Embeddings (Rotary)
Dulmar
This elegant trick lets transformers understand relative distances and extend gracefully to longer contexts.
quusid qoto dheer
Transformers-yadu ma laha dareen ku dhisan nidaamka, marka waxay u baahan yihiin macluumaadka booska lagu daro si uun. Moodooyinka hore waxay ku darsadeen xididada sinusoidal go'an ama meelaynta booska la bartay Xadhig, oo ay soo jeediyeen Su iyo asxaabtiisa sanadka 2021, waxay qaadataa hab ka duwan: Halkii lagu dari lahaa vector booska, waxay ku rogtaa lamaanaha cabbirada su'aasha iyo fiilooyinka muhiimka ah iyada oo loo eegayo xagal ku koraya booska calaamadda. Marka moodeelku uu xisaabiyo badeecada dhibco u dhaxaysa su'aal ku taal booska m iyo furaha booska n, xisaabtu way shaqeysaa markaas natiijadu waxay kuxirantahay kaliya fogaantooda m laga jaray n. Tani waxay siinaysaa wacyigelin dhab ah oo qaraabo ah, waxay si wacan ugu ciyaartaa kernels feejignaanta oo hufan, waxayna u kharribaa dareenka si habsami leh iyadoo la fogaanayo. RoPE hadda waxaa lagu isticmaalaa Llama, Mistral, Qwen, iyo inta badan moodooyinka furan ee casriga ah.
Aragtida Farsamada
RoPE waxa uu daaweeyaa cabbirada isku xidhka laba-labo oo waxa uu ku dabaqaa wareeg 2D lammaane kasta, iyada oo lammaane kala duwani ay ku wareegayaan soo noqnoqoshada kala duwan, si la mid ah gacmaha saacado badan oo ku socda xawaare kala duwan. Sababtoo ah ku wareejinta booska m ka dibna qaadashada badeecada dhibicda wax lagu beddelay booska n waxay ka tagtaa farqiga xagasha oo kaliya, buundooyinka feejignaanta waxay noqdaan shaqooyinka booska qaraabo. Lammaanaha soo noqnoqda ee sarreeya waxay qabtaan nidaam maxalli ah oo wanaagsan; Lamaanayaasha soo noqnoqda hooseeya waxay qabtaan booska fogaanta. Muhiimad ahaan, waxay wax ka beddeshaa weydiimaha iyo furayaasha, ma beddelo qiyamka.
Saamaynta Istiraatijiyadeed
Xawaaraha iyo miisaanka
Socodka shaqada luqaddu si dhakhso leh ayay u socon kartaa iyada oo aan la hurayn joogteynta.
Helitaanka iyo gaarsiinta
Waxay balaadhisaa gelitaanka luqadaha iyo qaababka isgaarsiinta.
Go'aamo cad
Kooxuhu waxay waqti badan ku qaadan karaan xukunka halka otomaatiggu uu qabanayo ku celcelinta.
Mustaqbalka Isku-xidhka Booska Wareega
Shaqooyin badan oo dhowaan soo baxay ayaa diiradda saaraya u fidinta XIGASHA ilaa macnaha guud oo aad uga dheer qaabka lagu tababaray. Farsamooyinka sida isdhexgalka booska, NTK-ka warhaynta miisaanka, iyo YaRN waxay hagaajiyaan wareegyada wareegyada si moodeel lagu tababaray, dheh, 4K tokens wuxuu xamili karaa 32K ama ka badan oo leh hagaajin khafiif ah. Filo in RoPE uu sii ahaado nidaamka mawqifka ugu sarreeya, oo leh dib-u-habayn joogto ah oo lagu samaynayo soo noqnoqoshada salkeeda iyo cabbirida xaaladaha milyan-token, iyo sii wadida daraasadda sida ay ula falgasho hab-dhaqanka feejignaanta.
Dhaqangelinta Adduunka-dhabta ah
Siinta Llama, Mistral, iyo Qwen mooddo dareenkooda nidaamka calaamadaynta iyada oo aan meelna lagu dhejin
Kordhinta macnaha guud ee la isticmaali karo laga bilaabo dhawr kun ilaa tobanaan kun oo calaamado ah iyada oo la adeegsanayo interpolation ama YaRN
Caawinta moodooyinka koodka si ay ula socdaan masaafada u dhaxaysa xirmooyinka, shaqooyinka, iyo tixraacyada faylalka dhaadheer
Taageerida su'aalaha dukumeentiga dheer ee ka jawaabaya halka ay qaraabo yihiin oo u dhexeeya su'aasha iyo arrimaha caddaynta
Khatarta & Dariiqyada Ilaalada
Xaqiiqooyinka dhalanteed waxay si deggan u geli karaan warbixinnada, taageerada socodka, ama natiijooyinka cilmi-baarista.
Dareenka degdega ahi wuxuu abuuri karaa natiijooyin aan iswaafaqayn codsiyada la midka ah.
Xogta qoraalka xasaasiga ah ayaa laga yaabaa in la kashifo haddii kontaroolada gelitaanka ay daciif yihiin.
Qorshe Hawleedka Dhaqangelinta
Qeex qaabka wax soo saarka, codka, iyo heerarka tayada ka hor inta aan la baahin.
Jawaabaha salka ku haya ilo lagu kalsoon yahay mar kasta oo saxnidu ay muhiim tahay.
Hayso isbaarada dib u eegista bini aadamka ee wax soo saarka sare.
Lasoco qaababka guuldarada oo dib u leyli dardargelinta ama socodka shaqada si joogto ah.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Rotary Position Embeddings quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
Eexda booska ALiBi
Su'aalaha soo noqnoqda
What is Rotary Position Embeddings?
Rotary Position Embeddings (Rotary Position Embeddings (Rotary) Khiyaamadan quruxda badan waxay u ogolaataa transformers-ku inay fahmaan masaafooyinka u dhexeeya oo ay si qurux badan u fidiyaan xaaladaha dhaadheer.
Sidee buu RoPE ugu duraa macluumaadka booska beddelka?
RoPE waxa uu ku wareejaa weydiinta iyo fallaadhaha muhiimka ah xagal u dhiganta booska, halkii uu ku dari lahaa boos gooni ah.
Qaybahee ka mid ah xisaabinta dareenka ayuu RoPE wax ka beddelaa?
Xadhiggu waxa uu ku dabaqaa wareegyadiisa su'aasha iyo fallaadhaha muhiimka ah, isaga oo ka tagaya qiyamka vectors lama taaban.
Waa maxay sababta lammaaneyaasha cabbirka kala duwan ay ugu wareegaan inta jeer ee RoPE?
Sida gacmaha saacada oo xawli kala duwan ku socda, soo noqnoqoshooyin kala duwan ayaa u oggolaanaya lammaaneyaasha qaarkood inay codeeyaan nidaamka muddada-gaaban iyo kuwa kale ee booska fogfog.
Waa maxay yoolka farsamooyinka sida isdhexgalka booska, NTK-ka warhaynta, iyo YaRN?
Hababkani waxay dib u cabbiraan wareegyada wareegga RoPE si uu moodelku u maareeyo duruufo aad uga dheer marka loo eego dhererka tababarka asalka ah.