Matsayin Interpolation don Dogon Magana
Matsakaicin Matsayi (PI) wata hanya ce mai sauƙi, mai tasiri wacce ke faɗaɗa taga mahallin Transformer ta hanyar matse sabbin fihirisar matsayi a cikin kewayon samfurin ya rigaya ya sani.
Dubawa
Instead of extrapolating to unseen positions, it interpolates within trained ones, requiring only brief fine-tuning.
Zurfafa nutsewa
Masu bincike Meta ne suka gabatar da su (Chen et al.) a cikin 2023, Matsayin Interpolation yana magance gaskiyar cewa samfura tare da RoPE suna kasawa da bala'i lokacin da aka fitar da su zuwa matsayi fiye da horo. Hankali ya sabawa: maimakon tambayar ƙirar don ɗaukar ƙimar matsayi mafi girma wanda bai taɓa gani ba, PI tana raba fihirisar matsayi mai shigowa ta hanyar ma'auni don haka tsawon maƙasudi, ka ce, taswirorin 8K baya cikin kewayon 2K na asali. Saboda an horar da ƙirar akan wannan kewayon, jujjuyawar suna zama cikin rarrabawa. Bayan kawai matakan daidaitawa 1,000, samfurin LLAMA ya tsawaita wannan hanyar da aka sarrafa har zuwa mahallin 32K. Takardar ta nuna cewa zazzagewa na iya busar da ma'aunin hankali zuwa manyan ƙima, yayin da interpolation ke riƙe su da iyaka da kwanciyar hankali, wanda shine dalilin da ya sa interpolation ke aiki da kyau fiye da ƙari.
Fahimtar Fasaha
PI tana sake daidaita matsayi m zuwa m/s inda s shine ma'aunin tsawo (misali, sabon tsayin da aka raba da tsayin asali). Don RoPE wannan yana rage girman matakin jujjuyawar tsakanin wurare kusa, tattara ƙarin matsayi cikin kewayon kusurwar da aka horar. Ƙididdigar ƙa'idar da ke cikin takarda tana nuna ƙimar kulawar da aka haɗa tare da kulawa da kyau, yayin da rashin ƙarfi na iya haifar da oda mai girma fiye da duk abin da aka gani a horo, yana lalata softmax.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Matsayin Matsayi don Dogon Mahimmanci
Matsakaicin Matsayi ya zama tushe don ɗimbin bin diddigin, gami da NTK-aware scaling da YaRN, waɗanda ke shiga cikin zaɓaɓɓu don adana cikakkun bayanai na gida. Hanya ita ce zuwa hanyoyin da ke buƙatar kaɗan ko babu gyarawa kuma zuwa ga yin gasa dogon yanayi a cikin horarwa. PI ya kasance tushe mai mahimmanci kuma galibi ana haɗe shi tare da sabbin tsare-tsare-tsare-tsare don isa 128K-da windows mahallin da inganci.
Aiwatar da Gaskiyar Duniya
Ƙaddamar da samfurin LLAMA na yanayi na 2K don ɗaukar alamun 8K-32K tare da matakan daidaitawa kusan 1,000
Daidaita samfurin taɗi na yanzu don taƙaitaccen takaddun bayanai ba tare da sake horarwa daga karce ba
Yin aiki azaman tushen tushen ra'ayi wanda NTK-sane da sikelin da YaRN suka inganta akan
Ƙaddamar da lambar yanayi mai tsawo ko bincike-takardun doka akan ƙirar da aka horar da su da gajerun tagogi
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Positional Interpolation for Long Context quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Matsayin Interpolation don Tsawaita Magana
Tambayoyin da ake yawan yi
What is Positional Interpolation for Long Context?
Matsakaicin Matsayi (PI) wata hanya ce mai sauƙi, mai tasiri wacce ke faɗaɗa taga mahallin Transformer ta hanyar matse sabbin fihirisar matsayi a cikin kewayon samfurin ya rigaya ya sani. Maimakon yin gyare-gyare zuwa wuraren da ba a gani ba, yana shiga tsakanin waɗanda aka horar da su, yana buƙatar taƙaitaccen daidaitawa kawai.
Menene ainihin ra'ayin da ke bayan Matsayin Interpolation?
PI yana raba fihirisar matsayi ta hanyar ma'auni don haka taswirorin da suka daɗe suna komawa cikin kewayon matsayi da aka horar, yana kiyaye juyawa cikin rarrabawa.
Me yasa fitar butulci zuwa matsayi masu tsayi ya gaza?
Takardar PI ta nuna manyan matsayi da ba a gani ba na iya samar da matakan kulawa da girman girman horo, lalata softmax.
Kusan nawa nawa aikin PI na asali ya buƙaci tsawaita LLAMA zuwa 32K?
Takardar ta ba da rahoton cewa kusan matakan daidaitawa 1,000 sun isa don tsawaita mahallin har zuwa alamun 32K.
Idan kun tsawaita mahallin ta hanyar s, ta yaya PI ke canza matsayi m?
PI tana sake daidaita matsayi m zuwa m/s, yana matsawa sabon kewayo mafi girma cikin kewayon horarwa na asali.
Ta yaya PI ta yi tasiri a hanyoyin da suka biyo baya kamar YaRN?
PI kafa interpolation a matsayin amintacciyar hanya; NTK-sane da sikelin da YaRN sun tace shi ta hanyar haɗa mitoci da zaɓin zaɓi.