Ibirindiro byumwanya
Ibirindiro byumwanya (RoPE) kodegisi aho buri kimenyetso cyicaye murukurikirane mukuzenguruka ikibazo cyacyo hamwe nurufunguzo rwibanze ku mpande zingana n'umwanya.
Incamake
This elegant trick lets transformers understand relative distances and extend gracefully to longer contexts.
Kwibira cyane
Abahindura ntabwo bafite uburyo-bwuburyo bwo gutondekanya, bityo bakeneye amakuru yimyanya yongeweho muburyo runaka. Moderi yambere yongeyeho sinusoidal vectors cyangwa yize imyanya yashyizwe mubisubizo. RoPE, yatanzwe na Su na bagenzi be mumwaka wa 2021, ifata ubundi buryo: aho kongeramo icyerekezo cyumwanya, kizunguruka ibice bibiri mubibazo hamwe nibice byingenzi byerekeranye ninguni ikura hamwe nikimenyetso. Iyo icyitegererezo kibara akadomo hagati yikibazo kumwanya m nurufunguzo kumwanya n, imibare irakora kuburyo ibisubizo biterwa gusa nintera igereranije m ukuyemo n. Ibi bitanga isano nyayo-imyanya yo kumenya, ikina neza hamwe nintungamubiri zitaweho neza, kandi ikangirika neza hamwe nintera. RoPE ubu ikoreshwa muri Llama, Mistral, Qwen, hamwe na moderi igezweho igezweho.
Ubushishozi
RoPE ifata ibipimo byo gushyiramo ibice bibiri kandi ikoresha 2D kuzunguruka kuri buri jambo, hamwe na joriji zitandukanye zizunguruka kuri radiyo zitandukanye, cyane nkamaboko yamasaha menshi yikubita kumuvuduko utandukanye. Kuberako kuzunguruka kumwanya m hanyuma ugafata akadomo k'ibicuruzwa hamwe nikintu kizunguruka kumwanya n gisiga gusa itandukaniro ryinguni, amanota yo kwitondera ahinduka imikorere yumwanya ugereranije. Byinshi-byombi bifata gahunda nziza yaho; buke-buke buke bifata umwanya muremure. Byibanze, ihindura ibibazo nurufunguzo, ntabwo ari indangagaciro.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza Kuzenguruka Umwanya wo Kwinjiza
Ibikorwa byinshi biherutse kwibanda ku kurambura RoPE kurwego rurerure kuruta icyitegererezo cyahuguwe. Ubuhanga nka interpolation yimyanya, gupima NTK-kumenya, hamwe na YaRN ihindura imirongo yizunguruka kugirango moderi yatojwe, tuvuge, ibimenyetso 4K bishobora gukora 32K cyangwa birenga hamwe no guhuza neza. Tegereza RoPE gukomeza kuba gahunda yimyanya myanya, hamwe nogukomeza kunonosora inshuro zayo fatizo no gupima miriyoni-token, kandi ukomeze kwiga uburyo ikorana nimyitwarire yo kwitabwaho.
Gushyira mu bikorwa Isi
Gutanga Llama, Mistral, na Qwen byerekana uburyo bwabo bwo gutondekanya ibimenyetso nta gushyiramo imyanya itandukanye
Kwagura imiterere ikoreshwa yicyitegererezo kuva ku bihumbi bike kugeza ku bihumbi mirongo ibimenyetso ukoresheje interpolation cyangwa YaRN
Gufasha kode yerekana ikigereranyo kiri hagati yinyuguti, imikorere, hamwe na dosiye ndende
Gushyigikira inyandiko-ndende ikibazo gisubiza aho umwanya ugereranije hagati yibibazo nibimenyetso bifatika
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Rotary Position Embeddings quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
ALiBi Umwanya Kubogama
Ibibazo bikunze kubazwa
What is Rotary Position Embeddings?
Ibirindiro byumwanya (RoPE) kodegisi aho buri kimenyetso cyicaye murukurikirane mukuzenguruka ikibazo cyacyo hamwe nurufunguzo rwibanze ku mpande zingana n'umwanya. Ubu buriganya bwiza butuma abahindura bumva intera igereranijwe kandi bakagura neza murwego rurerure.
Nigute RoPE itera amakuru yimyanya muri transformateur?
RoPE izunguruka ikibazo nurufunguzo rwibanze ku mpande zingana n'umwanya, aho kongeramo imyanya itandukanye.
Nibihe bice byo kwitondera kubara RoPE ihindura?
RoPE ikoresha kuzenguruka kubibazo hamwe nurufunguzo rwibanze, hasigara agaciro indangagaciro zidakozwe.
Ni ukubera iki ibice bitandukanye byizunguruka kuri radiyo zitandukanye muri RoPE?
Kimwe n'amaboko yisaha yikubita ku muvuduko utandukanye, imirongo itandukanye ireka ibice bibiri bigashyiraho gahunda ngufi hamwe nibindi birebire.
Niyihe ntego yubuhanga nka interpolation yimyanya, gupima NTK-kumenya, na YaRN?
Ubu buryo bukuraho imirongo ya RoPE kugirango izenguruke kuburyo icyitegererezo gishobora gukora ibintu birebire kuruta uburebure bwumwitozo wambere.