Ukuhunyushwa Kwesikhundla Komongo Omude
I-Positional Interpolation (PI) iyindlela elula, enethonya enweba iwindi lomongo we-Transformer ngokucindezela izinkomba zendawo entsha kububanzi imodeli esiyazi kakade.
Uhlolojikelele
Instead of extrapolating to unseen positions, it interpolates within trained ones, requiring only brief fine-tuning.
I-Deep Dive
Yethulwe abacwaningi be-Meta (Chen et al.) ngo-2023, I-Positional Interpolation ibhekana neqiniso lokuthi amamodeli ane-RoPE ahluleka ngendlela eyingozi uma edlulela ezikhundleni ezingaphezu kokuqeqeshwa. Ukuqonda kuphambene: kunokucela imodeli ukuthi isingathe amanani ezindawo amakhulu engakaze iwabone, i-PI ihlukanisa izinkomba zendawo engenayo ngesikali ukuze ubude obuqondiwe, okuthi, 8K bubuyele kububanzi bangempela be-2K. Ngenxa yokuthi imodeli yaqeqeshwa kulolo bubanzi, ukuzungezisa kuhlala kusatshalaliswa. Ngemuva kwezinyathelo zokushuna kahle eziyi-1,000 nje kuphela, imodeli ye-LLaMA inwetshwe ngale ndlela isingathwe kufika kumongo ongu-32K. Iphepha libonise ukuthi i-extrapolation ingaqhumisa amaphuzu wokunaka kumanani amakhulu, kuyilapho ukuhumusha kugcine kuboshwe futhi kuzinzile, yingakho ukuhumusha kusebenza kangcono kakhulu kunokwengeza.
I-Technical Insight
I-PI ilinganisa kabusha indawo kusuka ku-m kuya ku-m/s lapho u-s eyisici sokunweba (isb., ubude obusha buhlukaniswe ngobude bangempela). Nge-RoPE lokhu kusifinyeza ngempumelelo isinyathelo sokuphenduka phakathi kwezindawo eziseduze, kupakishe izindawo eziningi ebangeni le-angular eliqeqeshiwe. Ukuboshelwa kwethiyori ephepheni kukhombisa ukuthi amaphuzu okunaka ahlanganisiwe ahlala elawulwa kahle, kanti i-naive extrapolation ingaveza ama-oda wamaphuzu amakhulu kunanoma yini ebonwa ekuqeqesheni, ephazamisa i-softmax.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Lokuhunyushwa Kwesikhundla Kokuqukethwe Okude
I-Positional Interpolation ibe yisisekelo segagasi lokulandela, okuhlanganisa ukukala kwe-NTK-aware kanye ne-YaRN, ehumusha ngokukhetha kakhudlwana ukuze kulondolozwe imininingwane yendawo. I-trajectory ibheke ezindleleni ezidinga ukulungiswa okuncane noma ezingasho lutho futhi zibheke ekubhakeni ukuphatha kokuqukethwe okude ekuqeqesheni kwangaphambili. I-PI ihlala iyisisekelo esibalulekile futhi ivamise ukuhlanganiswa nezikimu ezintsha zokuqaphela imvamisa ukuze ifinyelele umongo ongu-128K-plus windows kahle.
Ukuqaliswa Komhlaba Wangempela
Ukunweba imodeli ye-LLaMA yokuqukethwe okungu-2K ukuze kuphathwe amathokheni e-8K-32K ngezinyathelo zokushuna kahle ezingaba ngu-1,000
Ukujwayela imodeli yengxoxo ekhona yokufingqa idokhumenti ende ngaphandle kokuqeqeshwa kabusha kusukela ekuqaleni
Isebenza njengesisekelo somqondo lapho ukukala kwe-NTK-aware kanye ne-YaRN kuthuthukisa khona
Ivumela ikhodi yomongo omude noma ukuhlaziywa kwedokhumenti yomthetho kumamodeli aqeqeshwe ngamawindi amafushane
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Positional Interpolation for Long Context quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Position Interpolation for Context Extension
Imibuzo evame ukubuzwa
What is Positional Interpolation for Long Context?
I-Positional Interpolation (PI) iyindlela elula, enethonya enweba iwindi lomongo we-Transformer ngokucindezela izinkomba zendawo entsha kububanzi imodeli esiyazi kakade. Esikhundleni sokudlulela ezindaweni ezingabonakali, ingena phakathi kwabaqeqeshiwe, idinga ukulungiswa okufushane nje.
Imuphi umqondo owumongo wePositional Interpolation?
I-PI ihlukanisa izinkomba zendawo ngesici sesikali ukuze imephu ilandelelane ibuyele ebangeni lendawo eqeqeshiwe, igcina ukuzungezisa kusatshalaliswe.
Kungani i-extrapolation engenangqondo ezikhundleni ezinde ihluleka?
Iphepha le-PI libonise izikhundla ezinkulu ezingabonakali zingakhiqiza ama-oda wamaphuzu wokunakwa amakhulu kunokuqeqeshwa, adilize i-softmax.
Cishe kungakanani ukulungisa okuhle okudingwe umsebenzi woqobo we-PI ukuze kunwetshwe i-LLaMA ibe ngu-32K?
Iphepha libike ukuthi cishe izinyathelo ze-1,000 zokulungisa kahle zanele ukunweba umongo kuze kufike kumathokheni angama-32K.
Uma unweba umongo nge-factor s, i-PI iyiguqula kanjani indawo m?
I-PI ilinganisela indawo ku-m ukuya ku-m/s, icindezela ububanzi obusha obukhudlwana kububanzi bangempela obuqeqeshiwe.
Ngabe i-PI yaba nomthelela kanjani ezindleleni zakamuva ezifana ne-YaRN?
I-PI isungule ukuhumusha njengendlela ephephile; Ukukala kwe-NTK-aware kanye ne-YaRN kwayilungisa ngokufaka amafrikhwensi ngokukhetha kakhulu.