Rukunin Matsayin Rotary
Rotary Position Embeddings (RoPE) yana ɓoye inda kowace alama ke zaune a jere ta hanyar jujjuya tambayar sa da maɓalli na maɓalli ta kusurwa daidai da matsayi.
Dubawa
This elegant trick lets transformers understand relative distances and extend gracefully to longer contexts.
Zurfafa nutsewa
Masu canji ba su da ma'anar tsari, don haka suna buƙatar ƙara bayanin matsayi ko ta yaya. Samfuran farko sun ƙara kafaffen vectors sinusoidal ko abubuwan da aka koya a cikin abubuwan da aka shigar. RoPE, wanda Su da abokan aiki suka gabatar a cikin 2021, yana ɗaukar wata hanya ta dabam: maimakon ƙara maƙasudin matsayi, yana juya nau'i-nau'i na nau'i a cikin tambaya da maɓalli masu mahimmanci ta kusurwar da ke girma tare da matsayi na alamar. Lokacin da ƙirar ta ƙididdige samfurin ɗigo tsakanin tambaya a matsayi m da maɓalli a matsayi n, lissafin yana aiki don haka sakamakon ya dogara ne kawai akan danginsu na nesa m rage n. Wannan yana ba da wayar da kan dangi na gaskiya, yana wasa da kyau tare da ingantaccen kernels na hankali, kuma yana lalata hankali sosai tare da nisa. Ana amfani da RoPE yanzu a Llama, Mistral, Qwen, da mafi yawan samfuran buɗaɗɗen zamani.
Fahimtar Fasaha
RoPE yana kula da nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan nau'ikan suna aiki da jujjuyawar 2D ga kowane nau'i-nau'i, tare da nau'ikan nau'ikan nau'ikan nau'ikan suna juyawa a mitoci daban-daban, kamar hannayen agogo da yawa suna yin la'akari da sauri daban-daban. Saboda juyawa ta matsayi m sannan ɗaukar samfurin ɗigo tare da wani abu mai juyawa ta matsayi n ya bar kawai bambancin kusurwa, ƙimar hankali ya zama ayyuka na matsayi na dangi. Maɗaukakin nau'i-nau'i suna ɗaukar tsari mai kyau na gida; ƙananan mitoci suna ɗaukar matsayi mai tsayi. Mahimmanci, yana canza tambayoyi da maɓalli, ba ƙima ba.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Rotary Matsayin Sakawa
Yawancin ayyuka na baya-bayan nan sun mayar da hankali kan shimfiɗa RoPE zuwa mahallin da ya fi tsayi fiye da yadda aka horar da samfurin a kai. Dabaru kamar interpolation matsayi, NTK-sane scaling, da YaRN daidaita jujjuya mitoci don haka samfurin horar da a kan, ce, 4K Alamu iya rike 32K ko fiye tare da haske kyau-daidaita. Yi tsammanin RoPE ya kasance babban tsarin matsayi, tare da ci gaba da gyare-gyare zuwa mitar tushe da ƙima don mahallin alamar miliyoyin, da kuma ci gaba da nazarin yadda yake hulɗa da halin kulawa.
Aiwatar da Gaskiyar Duniya
Bayar da Llama, Mistral, da Qwen suna ƙirƙira ma'anar tsarin alamar su ba tare da sanya wuri daban ba
Ƙirƙirar mahallin abin amfani da samfurin daga ƴan dubbai zuwa dubun dubatar alamun ta hanyar interpolation ko YaRN
Taimakon ƙirar ƙira don bibiyar nisa tsakanin maɓalli, ayyuka, da nassoshi cikin dogayen fayiloli
Taimakawa dogon takardan tambaya yana amsa inda matsayi na dangi tsakanin tambaya da abubuwan shaida
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Rotary Position Embeddings quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Matsayin ALiBi Bias
Tambayoyin da ake yawan yi
What is Rotary Position Embeddings?
Rotary Position Embeddings (RoPE) yana ɓoye inda kowace alama ke zaune a jere ta hanyar jujjuya tambayar sa da maɓalli na maɓalli ta kusurwa daidai da matsayi. Wannan kyakkyawan dabarar yana ba da damar masu taswira su fahimci nisan dangi kuma su fadada cikin alheri zuwa mafi tsayi.
Ta yaya RoPE ke allurar bayanin matsayi a cikin na'ura?
RoPE yana jujjuya tambaya da maɓalli na maɓalli ta kwana daidai gwargwado, maimakon ƙara daɗaɗɗen vector na daban.
Wadanne sassa na lissafin hankali ne RoPE ke gyara?
RoPE yana amfani da jujjuyawar sa zuwa tambayar da maɓalli na maɓalli, yana barin ƙimar ƙimar ba a taɓa shi ba.
Me yasa nau'i-nau'i nau'i-nau'i daban-daban suke juyawa a mitoci daban-daban a cikin RoPE?
Kamar hannayen agogo suna yin tikitin gudu daban-daban, mitoci daban-daban suna barin wasu nau'i-nau'i su ɓoye tsari na gajeriyar hanya da sauran matsayi mai tsayi.
Menene makasudin dabaru kamar interpolation matsayi, NTK-sane da sikelin, da YaRN?
Waɗannan hanyoyin suna sake daidaita mitocin jujjuyawar RoPE don haka ƙira zai iya ɗaukar yanayi mai tsayi fiye da tsayin horonsa na asali.