I-Position Interpolation for Context Extension
I-Position Interpolation (PI) iwuhlelo olunweba iwindi lomongo elisebenzisekayo lemodeli yolimi lidlulele kude nobude balo bokuqeqeshwa ngokukhipha izinkomba zendawo esikhundleni sokuzihlanganisa.
Uhlolojikelele
It lets a model trained on, say, 2K or 4K tokens handle 32K or more with only light fine-tuning.
I-Deep Dive
Ama-LLM amaningi esimanje asebenzisa ukushumekwa kwe-rotary positional (RoPE), okuhlanganisa indawo njengama-engeli ajikelezayo asetshenziswa embuzweni kanye namavekhtha angukhiye. Uma umane uphakele ukulandelana okude, imodeli ibona izindawo kanye nama-engeli aphendukayo engakaze iqeqesheke kuwo, futhi ukusebenza kuyawohloka ngenxa yokuthi ukunaka kudlulela kabi kumafrikhwensi angaphandle kwebanga. I-Position Interpolation igwema i-extrapolation: ukwelula ukusuka kubude L ukuya kubude L', ihlukanisa inkomba ngayinye yendawo nge-factor L'/L, icindezela ububanzi obusha bubuyele esikhaleni esiqeqeshiwe. Imodeli manje ike ibone kuphela ama-engeli okusabalalisa, avele ahlukaniswe kakhulu. Ukushuna kahle okufushane (ngokuvamile okuyizinyathelo ezingamakhulu ambalwa ukuya kweziyinkulungwane) kuyivumela ukuthi ivumelane nesikhala esikahle, inikeze ukuziphatha kokuqukethwe okuzinzile kokuqukethwe okude ngengxenye encane yezindleko zokuqeqesha kusengaphambili.
I-Technical Insight
I-RoPE izungezisa amapheya anobukhulu kumafrikhwensi ahamba kahle ukuya emaholoholo. I-PI iphinda isuse indawo ku-m iye ku-m/s lapho s = L'/L, ngakho-ke ama-engeli azungezisayo ahlala phakathi kwebanga eliqeqeshiwe kunokudlulela phambili. Izinhlobonhlobo eziqaphela imvamisa ezifana ne-NTK-aware scaling kanye ne-YaRN ziya phambili: zikala amaza aphansi kancane futhi amaza aphezulu kakhulu (noma zihlanganise ngobude be-wavelength), zilondoloze imininingwane yendawo yemvamisa ephezulu kuyilapho inweba ukufinyelela ebangeni elide le-low-frequency.
I-Strategic Impact
Isivinini nesikali
Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.
Finyelela futhi ufinyelele
Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.
Izinqumo ezicacile
Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.
Ikusasa Lokuhunyushwa Kwesikhundla Sesandiso Sokuqukethwe
Isandiso somongo sihamba ngokushesha. Izindlela ezifana ne-NTK-aware RoPE scaling, YaRN, kanye ne-dynamic/long-RoPE manje isunduza amafasitela kumakhulu ezinkulungwane noma izigidi zamathokheni, ngezinye izikhathi anokucutshungulwa okuncane noma angekho nhlobo. Lindela lawa maqhinga okukala ukuthi ahlanganiswe nokunaka okusebenzayo kanye nokucindezelwa kwenqolobane ye-KV, futhi abe amafindo ajwayelekile ekucushweni kwemodeli. Ucwaningo luyaqhubeka ekugcineni ukunemba kuphezulu kulo lonke iwindi eligcwele ukuze okuqukethwe okude kusetshenziswe ngempela, hhayi nje ukusekelwa ngegama.
Ukuqaliswa Komhlaba Wangempela
Ukunweba imodeli ye-LLaMA eqeqeshwe nge-4K ibe mongo we-32K ukuze kufinyezwe amadokhumenti amade ngemva kokulungisa kafushane.
Ilayisha yonke i-codebase noma inkontileka enkulu yomthetho ekwazisweni okukodwa kokuphendulwa kwemibuzo ehlukahlukene.
Ukusebenzisa i-NTK-aware noma i-YaRN scaling ukuze unwebe umongo ngokuqeqeshwa okuncane noma okungenakho okwengeziwe.
Ukukhonza imilando emide yengxoxo ngaphandle kokuncishiswa ngokuhlehlisa izikhundla ze-RoPE ngesikhathi sokunquma.
Izingozi & Guardrails
Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.
Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.
Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.
Ukuqalisa Umhlahlandlela
Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.
Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.
Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.
Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Position Interpolation for Context Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
Isandiso sobude be-YaRN kanye Nokuqukethwe
Imibuzo evame ukubuzwa
What is Position Interpolation for Context Extension?
I-Position Interpolation (PI) iwuhlelo olunweba iwindi lomongo elisebenzisekayo lemodeli yolimi lidlulele kude nobude balo bokuqeqeshwa ngokukhipha izinkomba zendawo esikhundleni sokuzihlanganisa. Ivumela imodeli eqeqeshwe, ithi, amathokheni angu-2K noma angu-4K aphathe i-32K noma ngaphezulu ngokulungiswa kahle okukhanyayo kuphela.
Uyini umqondo oyinhloko wePosition Interpolation?
I-PI ihlukanisa izinkomba zokuma ngesici sokunweba, ibeka imephu ukulandelana okude emuva kububanzi bokusabalalisa imodeli esivele yayifunda.
Isiphi isikimu sombhalo wekhodi esihlobene kakhulu ne-Position Interpolation?
I-PI yaduma kumamodeli asuselwa ku-RoPE, ikhulisa ama-engeli ajikelezayo ukuze ahlale ngaphakathi kwamafrikhwensi aqeqeshiwe.
Kungani i-extrapolation engenangqondo emongweni omude ivame ukwehluleka?
Ukondla ukulandelana okude kuveza imodeli kuma-engeli angaphandle kobubanzi engazange iqeqeshwe kuwo, okubangela ukunaka nokusebenza kwehle kakhulu.
Uma kunwetshwa ubude bomongo ukusuka ku-L kuye ku-L', iyiphi isici sokukala kabusha i-PI eyisisekelo esebenza ku-position m?
I-PI ibeka imephu ku-m ukuya ku-m ihlukaniswa yi-factor s = L'/L, icindezela ibanga elide libe isikhawu soqobo esiqeqeshiwe.
Ngabe ukukala kwe-NTK-aware kanye ne-YaRN kuthuthuka kanjani ku-plain Position Interpolation?
Lezi zindlela zikala amaza aphansi naphezulu ngendlela ehlukile, zigcina imininingwane yendawo ecolisekile ngenkathi zisafinyelela okuqukethwe okude.