Matsayin Interpolation don Tsawaita Magana
Matsayin Interpolation (PI) wata dabara ce da ke shimfiɗa tagar mahallin da za a iya amfani da samfurin harshe sama da tsawon horonsa ta hanyar sake daidaita fihirisar matsayi maimakon fitar da su.
Dubawa
It lets a model trained on, say, 2K or 4K tokens handle 32K or more with only light fine-tuning.
Zurfafa nutsewa
Yawancin LLMs na zamani suna amfani da abubuwan sakawa na rotary (RoPE), waɗanda ke ɓoye matsayi azaman kusurwoyin juyawa da ake amfani da su ga tambaya da maɓalli. Idan kawai kuna ciyar da jerin tsayin daka, ƙirar tana ganin matsayi da kusurwoyi na juyawa wanda bai taɓa horar da su ba, kuma aikin ya ruguje saboda hankali ya ƙaru zuwa mitoci marasa iyaka. Matsayin Interpolation yana guje wa haɓakawa: don tsawanta daga tsayin L zuwa tsayin L', yana raba kowane ma'aunin matsayi da ma'aunin L'/L, yana matse sabon kewayo zuwa cikin tazarar horo. Samfurin yanzu ya taɓa ganin kusurwoyin rarrabawa kawai, kawai an raba shi da yawa. Shortarancin sauti mai kyau (sau da yawa ƴan matakai ɗari zuwa dubu) yana ba shi damar daidaitawa zuwa mafi kyawun tazara, yana haifar da tsayayyen hali mai tsayi a ɗan ƙaramin juzu'i na ƙimar horo.
Fahimtar Fasaha
RoPE yana jujjuya nau'i-nau'i nau'i-nau'i a mitoci waɗanda ke da kyau zuwa ƙanƙara. PI yana sake daidaita matsayin m zuwa m/s inda s = L'/L, don haka kusurwar juyawa ta kasance cikin kewayon da aka horar da su maimakon cirewa. Bambance-bambancen sane da mitoci kamar NTK-aware scaling da YaRN sun ci gaba: suna haɓaka ƙananan mitoci kaɗan da ƙarin mitoci masu yawa (ko tsaka-tsaki da tsayin raƙuman ruwa), suna adana cikakkun bayanai na gida na mitoci masu tsayi yayin da suke ƙara ƙarancin mitoci mai nisa.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Matsayin Matsayi don Tsawaita Mahimmanci
Tsawaita yanayin yana tafiya da sauri. Hanyoyi irin su NTK-aware RoPE scaling, YaRN, da dynamic/dogon-RoPE yanzu suna tura windows zuwa daruruwan dubban ko ma miliyoyin alamu, wani lokaci tare da kadan ko babu gyarawa. Yi tsammanin waɗannan dabaru na sikeli za a haɗa su tare da ingantaccen kulawa da matsi na KV-cache, kuma su zama madaidaicin ƙulli a cikin saitunan ƙira. Bincike yana ci gaba da kiyaye daidaito a saman cikakken taga don haka dogayen abubuwan ana iya amfani da su da gaske, ba wai kawai ana goyan bayan su ba.
Aiwatar da Gaskiyar Duniya
Ƙarfafa samfurin LLAMA da aka horar da 4K zuwa mahallin 32K don taƙaita dogayen takardu bayan taƙaitaccen daidaitawa.
Loda gaba ɗaya lambar tushe ko babban kwangilar doka cikin hanzari ɗaya don amsa tambayoyin fayil ɗin giciye.
Yin amfani da sikelin NTK-aware ko YaRN don tsawaita mahallin tare da ƙaramin ko babu ƙarin horo.
Hidimar dogon tarihin taɗi ba tare da yanke ba ta hanyar sake fasalin matsayin RoPE a lokacin ƙididdigewa.
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Position Interpolation for Context Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
YaRN da Tsawon Tsawon yanayi
Tambayoyin da ake yawan yi
What is Position Interpolation for Context Extension?
Matsayin Interpolation (PI) wata dabara ce da ke shimfiɗa tagar mahallin da za a iya amfani da samfurin harshe sama da tsawon horonsa ta hanyar sake daidaita fihirisar matsayi maimakon fitar da su. Yana ƙyale samfurin da aka horar da shi, a ce, alamun 2K ko 4K suna ɗaukar 32K ko fiye tare da daidaitawa mai haske kawai.
Menene ainihin ra'ayin Matsayi Interpolation?
PI tana raba fihirisar matsayi ta hanyar haɓakawa, tana yin taswirar mafi tsayi a cikin kewayon rarrabawa samfurin da aka rigaya ya koya.
Wanne makircin rufaffen matsayi ya fi alaƙa da Matsayi Interpolation?
PI ya shahara don ƙirar tushen RoPE, yana sake daidaita kusurwoyin juyi don su kasance cikin mitoci da aka horar.
Me yasa fitar butulci zuwa dogon mahallin yakan gaza?
Ciyar da jeri mai tsayi yana fallasa ƙirar zuwa kusurwoyi marasa iyaka da ba a taɓa horar da su ba, yana haifar da hankali da aiki don raguwa sosai.
Idan tsawaita tsayin mahallin daga L zuwa L', menene ma'anar sake fasalin PI na asali ya shafi matsayi m?
PI taswirori m zuwa m raba ta hanyar factor s = L'/L, matsawa tsayin iyaka zuwa ainihin tazarar horarwa.
Ta yaya NTK-sane da sikelin da YaRN ke inganta akan madaidaicin Matsayi?
Waɗannan hanyoyin suna yin girman ƙananan ƙananan mitoci daban-daban, suna riƙe da cikakkun bayanai na matsayi na gida yayin da har yanzu suke kaiwa ga mafi tsayi.