Position Interpolation for Context Extension
Position Interpolation (PI) inzira inotambanudza hwindo rinoshandiswa remutauro wemodhiyo kupfuura kureba kwayo kwekudzidziswa nekudzoreredza ma indices enzvimbo panzvimbo pekuawedzera.
Pfupiso
It lets a model trained on, say, 2K or 4K tokens handle 32K or more with only light fine-tuning.
Kudzika Kwakadzika
Mazhinji maLLM emazuva ano anoshandisa rotary positional embeddings (RoPE), iyo inoisa chinzvimbo semakona ekutenderera anoiswa kumubvunzo nemakiyi mavheji. Kana iwe ukangodyisa kutevedzana kwakareba, modhi inoona zvinzvimbo nemakona ekutenderera zvayasina kumbodzidzira pairi, uye kuita kunodonha nekuti kutarisisa kunowedzera zvisina kunaka kune kunze-kwe-renji ma frequency. Position Interpolation inodzivirira kuwedzera: kuwedzera kubva paurefu L kusvika pakureba L', inokamura nzvimbo yega yega indekisi nechinhu L'/L, ichidzvanya mutsara utsva kudzokera mukati menguva yakadzidziswa. Iyo modhi ikozvino inongoona mu-kugovera angles, yakangopatsanurwa zvakanyanya. Kurongeka kupfupi (kazhinji nhanho mazana mashoma kusvika kuchuru) kunoita kuti ienderane nepakatsetseka, ichipa maitiro akatsiga enguva refu pachikamu chidiki chemutengo wepretraining.
Technical Insight
RoPE inotenderedza dimension pairs pamafrequency anotambanuka kusvika pakukora. PI inodzoreredza chinzvimbo m kusvika m/s uko s = L'/L, saka makona ekutenderera anogara mukati mechikamu chakadzidziswa kwete extrapolating. Frequency-aware siyana senge NTK-inoziva kuyera uye YaRN inoenda mberi: inoyera yakaderera frequency uye yakakwira ma frequency akawanda (kana interpolate newavelength), kuchengetedza yakakwirira-frequency yenzvimbo uku ichiwedzera yakaderera-frequency kureba-kusvika.
Strategic Impact
Kumhanya uye chiyero
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana.
Svika uye svika
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana.
Sarudzo dzakajeka
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora.
Ramangwana rePosition Interpolation yeContext Extension
Context extension iri kufamba nekukurumidza. Nzira dzakadai seNTK-inoziva RoPE kuyera, YaRN, uye inoshanduka/yakareba-RoPE ikozvino inosundira mahwindo kumazana ezviuru kana kunyange mamirioni ezviratidzo, dzimwe nguva nekushomeka kana kusanyatsogadzirisa. Tarisira kuti matipi ekuyera aya abatanidzwe nekunyatsotarisisa uye KV-cache compression, uye kuve akajairwa mapfundo mumodhi configs. Tsvagiridzo inoenderera mberi nekuchengetedza kurongeka kwakakwira pahwindo rakazara zvekuti mamiriro akareba anogona kunyatso shandiswa, kwete kungotsigirwa nemazita.
Real-World Implementation
Kuwedzera 4K-yakadzidziswa LLaMA modhi kune 32K mamiriro ekupfupisa magwaro marefu mushure menguva pfupi yekunyatsogadzirisa.
Kurodha kodhesi yese kana kondirakiti huru yepamutemo mune imwe yekuchimbidza yekupindura mubvunzo wemuchinjika.
Uchishandisa NTK-kuziva kana YaRN kuyera kurebesa mamiriro ane kushoma kana kusava nekuwedzera kudzidziswa.
Kushandira nhoroondo refu dzekutaura pasina kudzikiswa nekudzoreredza zvinzvimbo zveRoPE panguva yekufungidzira.
Njodzi & Guardrails
Chokwadi chehuroyi chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana tsvakiridzo.
Kunzwa nekukasira kunogona kugadzira mhedzisiro isingaenderane pane zvikumbiro zvakafanana.
Sensitive text data inogona kuburitswa kana zvidhiraivho zvisina kusimba.
Implementation Roadmap
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Position Interpolation for Context Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
YaRN uye Context Length Kuwedzerwa
Mibvunzo inowanzo bvunzwa
What is Position Interpolation for Context Extension?
Position Interpolation (PI) inzira inotambanudza hwindo rinoshandiswa remutauro wemodhiyo kupfuura kureba kwayo kwekudzidziswa nekudzoreredza ma indices enzvimbo panzvimbo pekuawedzera. Iyo inobvumira modhi yakadzidziswa, toti, 2K kana 4K tokens kubata 32K kana kupfuura nekungori nyore kugadzirisa.
Ndeipi pfungwa huru yePosition Interpolation?
PI inokamura maindisi echinzvimbo nechinhu chekuwedzera, ichigadzira kutevedzana kwakareba kudzoka muchikamu chekugovera iyo modhi yakatodzidzwa.
Ndeipi positional encoding scheme iri Position Interpolation inonyanya kuenderana nayo?
PI yakave yakakurumbira kune RoPE-yakavakirwa modhi, kudzoreredza makona ekutenderera kuitira kuti arambe ari mukati menguva dzakadzidziswa.
Sei kusaziva kwekuwedzera kune yakareba mamiriro kunowanzotadza?
Kudyisa kutevedzana kwakareba kunofumura modhi kumakona ekunze kwayasina kumbodzidzira, zvichikonzera kutarisisa uye kuita kuderera zvakanyanya.
Kana uchiwedzera kureba kwechirevo kubva kuL kuenda kuL', chii chinodzoreredza chinhu chinoshandiswa nePI yakakosha pachinzvimbo m?
PI mepu m kusvika m yakapatsanurwa nechinhu s = L'/L, ichidzvanya kureba muchikamu chepakutanga chakadzidziswa.
Ko NTK-inoziva kuyera uye YaRN inovandudza sei pane yakajeka Position Interpolation?
Idzi nzira dzinoyera dzakaderera uye dzepamusoro frequency zvakasiyana, dzichichengeta yakatsetseka-yakaomeswa yenzvimbo yenzvimbo ichiri kusvika kureba.