Rotary Position
Embeddings Position Rotary (RoPE) dafay kode fi token bu nekk di toog ci benn rang ci wëlbatiku laaj ak vecteur key ci angle bu méngoo ak position bi.
Résumé
This elegant trick lets transformers understand relative distances and extend gracefully to longer contexts.
Plongeur bu xóot
Transformatër yi amul benn gis-gis buñ tabax ci seen biir, moo tax dañu soxla leeral ci position bi ci anam wumu mëna doon. Modèle yu njëkk ya dañu daan yokk ay vecteur sinusoïdal yu fiks wala ñu jàng position ci biir duggal yi. RoPE, bi Su ak ay naataango tàmbale ci 2021, jël na beneen gis-gis: ludul yokk benn vecteur position, dafay wëlbati ñaari dimension ci laaj bi ak vecteur key yi ci angle buy màgg ak position token bi. Su model bi xaymaa produit dot bi ci digganté ab laajte ci position m ak ab caabi ci position n, math bi dafay dox suko defee resultaa bi a ngi aju ci seen distance relatif m dindi n. Loolu dafay joxe xam-xam bu dëggu ci position relatif, di jouer bu baax ak kernel yiy bàyyi xel, ba noppi di yàq bàyyi xel ci distance bi. Leegi ñu ngi jëfandikoo rope ci Llama, Mistral, Qwen ak ci model yu ubbeeku yi ci jamono jii.
Gis-gis xarala
RoPE dafay jëfandikoo dimension yuñ boole ci ñaar-ñaar, ba noppi def rotation 2D ci ñaar-ñaar bu nekk, ak ñaari-ñaar yu wuute yuy wëréelu ci frequence yu wuute, lu melni loxo montor yu bari yuy tic ci gaawaay yu wuute. Ndax wëlbatiku ci position m ba noppi jël produit dot ak luñu wëlbatiku ci position n du bàyyi ludul différence angle bi, poñ yi ñuy bàyyi xel dañuy nekk fonction position relatif. Peer yu am fréquence yu kawe yi dañuy jàpp doxalin bu baax ci gox bi; ñaari fréquence yu woyof yi dañuy jàpp position yu sori. Li gëna am solo mooy, dafay soppi laaj yi ak caabi yi, du valeur yi.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu position Rotary
Liggéey bu bari ci jamono yii mingi sëssé ci tàllal RoPE ci anam wu gëna gudd ci anam wi ñu ko tàggate. Pexe yu melni interpolation position, NTK-xam scaling, ak YaRN dañuy yamale fréquence rotation yi suko defee benn model buñu tàggat ci, wax, jeton 4K mën na jëfandikoo 32K wala lu ëpp ak ajustement bu woyof. Xaarandi RoPE ngir des ci xeetu position bi gëna am solo, ak ay coppite yuy wéy ci frequency base ak scaling ngir ay contexte milioŋ-token, ak wéyal jàngat ci ni muy doxalee ak jeffin juy bàyyi xel.
Doxal ci àdduna dëgg
Jox Llama, Mistral, ak Qwen ñu wane seen gis-gis ci njubluwaay bu amul benn position buñ tàqale
Yaatalal xeetu jëfandikoo gi ci ay junni yu néew dem ba ci ay fukki junni token jaaraleko ci interpolation wala YaRN
Dimbali modelu kod yi ñu mëna topp diggante parenthèses yi, fonction yi ak royuwaayi fichier yu gudd yi
Jàppale tontub laaj bu yàgg ci këyit, fu position relatif bi am ci digganté laaj ak firnde
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Rotary Position Embeddings quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
ALiBi position bias
Laaj yi ñuy faral di laaj
What is Rotary Position Embeddings?
Embeddings Position Rotary (RoPE) dafay kode fi token bu nekk di toog ci benn rang ci wëlbatiku laaj ak vecteur key ci angle bu méngoo ak position bi. Kaf gu taaru gii dafay may transformatër yi ñu xam distance relatif yi ba noppi ñu mëna dem ba ci contexte yu gëna gudd.
naka lay def ba RoPE dugal ay leeral ci position bi ci biir transformateur bi?
RoPE dafay wëlbati vecteur laaj ak caabi ci angle bu méngoo ak position bi, du yokk beneen vecteur position bu wuute.
Ban pàcc ci calcul attention la RoPE di soppi?
RoPE dafay jëfandikoo wëlbatiku yi ci laaj bi ak vecteur key yi, bàyyi vecteur valeur yi ñu laal leen.
Lan moo waral ñaari dimension yu wuute di wëréelu ci frequence yu wuute ci RoPE?
Daa melni loxo montor yiy tic ci gaawaay yu wuute, fréquence yu bari dañuy may yenn peer ñu kode rang bu gàtt, ñeneen ñi position bu sori.
Lan mooy mébetu pexe yu melni interpolaasioŋ position, eskalaasioŋ xam NTK, ak YaRN?
Pexe yooyu dañuy soppi frequence rotation RoPE yi suko defee model bi mëna jëflante ak contexte yu gëna gudd guddaayi tàggat yaram bimu njëkka am.