Positional Interpolation for Long Context
Positional Interpolation (PI) inzira yakapfava, ine pesvedzero inotambanudza hwindo reTransformer remukati nekudzvanya ma indices ezvinzvimbo muhuwandu iyo modhi inotoziva.
Pfupiso
Instead of extrapolating to unseen positions, it interpolates within trained ones, requiring only brief fine-tuning.
Kudzika Kwakadzika
Yakaunzwa ne Meta vaongorori (Chen et al.) muna 2023, Positional Interpolation inobata chokwadi chekuti mamodheru ane RoPE anokundikana zvakanyanya kana achiwedzera kune zvinzvimbo kupfuura kudzidziswa. Muono wacho unopokana: pane kukumbira modhi kuti ibate mahombe ezvinzvimbo zvayasati yamboona, PI inokamura maindisi enzvimbo anouya nechikero kuitira kureba kwe, toti, 8K mepu kudzoka muyekutanga 2K renji. Nekuti iyo modhi yakadzidziswa pane iyo siyana, iyo inotenderera inogara mu-kugovera. Mushure mekungoita chiuru-yekugadzirisa matanho, modhi yeLLaMA yakawedzera nzira iyi yakabatwa kusvika kumakumi matatu nemaviri mamiriro. Iro bepa rakaratidza kuwedzeredza kunogona kukwidza kutarisisa zvibodzwa kune hukuru hukuru, nepo kududzira kuchiita kuti irambe yakasungwa uye yakagadzikana, ndosaka kududzira kunoshanda zvakanyanya zvirinani pane kuwedzera.
Technical Insight
PI inodzoreredza nzvimbo m kusvika m/s uko s iri chinhu chekuwedzera (semuenzaniso, hurefu hutsva hwakakamurwa nehurefu hwepakutanga). KuRoPE izvi zvinonyatso dzikisa nhanho yekutenderera pakati pezvinzvimbo zvakapakana, kurongedza mamwe zvinzvimbo muakadzidziswa angular renji. Iyo theoretical yakasungwa mubepa inoratidza kutarisisa kwakaringana zvibodzwa zvinogara zvakadzorwa, nepo naive extrapolation inogona kuburitsa maodha ehukuru hwakakura kupfuura chero chinhu chinoonekwa mukudzidziswa, kudzikamisa softmax.
Strategic Impact
Mutengo uye bhajeti
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
Sarudzo dzakajeka
Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
Kudzora kwemhando yepamusoro
Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Ramangwana rePositional Interpolation for Long Context
Positional Interpolation yakave hwaro hwemafungu ekuteedzera, kusanganisira NTK-inoziva kuyera uye YaRN, iyo inodudzira zvakanyanya kusarudza kuchengetedza ruzivo rwenzvimbo. Iyo nzira ndeye nzira dzinoda kushoma kana kusanyatso gadzirisa uye kuenda kubheka-refu-mamiriro ekubata mupretraining. PI inoramba iri yakakosha yekutanga uye inowanzobatanidzwa neazvino frequency-aware zvirongwa zvekusvika 128K-pamwe nemamiriro ekunze windows zvinobudirira.
Real-World Implementation
Kuwedzera 2K-context LLaMA modhi yekubata 8K-32K tokens ine angangoita chiuru-yekugadzirisa matanho.
Kugadzirisa iyo iripo yekukurukura modhi yekupfupisa-refu-gwara pasina kudzidziswa kubva pakutanga
Kushanda seyambiro yekutanga iyo NTK-inoziva kuyera uye YaRN inovandudza pairi
Kugonesa kodhi-yakareba kodhi kana yemutemo-gwaro kuongororwa pamhando dzakadzidziswa pakutanga nemahwindo mapfupi
Njodzi & Guardrails
Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Implementation Roadmap
Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
Benchmark pasi pechokwadi mutoro uye data mamiriro.
Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Positional Interpolation for Long Context quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Position Interpolation for Context Extension
Mibvunzo inowanzo bvunzwa
What is Positional Interpolation for Long Context?
Positional Interpolation (PI) inzira yakapfava, ine pesvedzero inotambanudza hwindo reTransformer remukati nekudzvanya ma indices ezvinzvimbo muhuwandu iyo modhi inotoziva. Panzvimbo pokuwedzera kune zvisingaoneki, inopindirana mukati mevakadzidziswa, ichingoda kurongeka kwenguva pfupi.
Ndeipi pfungwa huru kumashure kwePositional Interpolation?
PI inokamura maindisi enzvimbo nechikero zvekuti mepu yakarebesa inoteedzana kudzokera muchikamu chakadzidziswa, ichichengeta kutenderera mukugovera.
Sei naive extrapolation kune nzvimbo refu ichitadza?
Iyo PI bepa yakaratidza zvisingaonekwe zvinzvimbo zvakakura zvinogona kuburitsa kutarisisa zvibodzwa zvehukuru kupfuura kudzidziswa, kudzikamisa softmax.
Zvingangoita kuti ingani-ikugadzirisa zvakanaka iyo yekutanga PI basa yaida kuwedzera LLaMA kusvika ku32K?
Pepanhau rakashuma kuti nhanho dzinokwana chiuru dzekugadzirisa zvakanaka dzaive dzakakwana kuwedzera mamiriro kusvika kumatokeni makumi matatu nemaviri.
Kana iwe ukawedzera mamiriro nechinhu s, PI inoshandura sei chinzvimbo m?
PI inodzoreredza chinzvimbo m kusvika m/s, ichidzvanya hutsva hukuru muyero yepakutanga yakadzidziswa.
PI yakapesvedzera sei nzira dzakazotevera seYaRN?
PI yakamisa kududzira senzira yakachengeteka; NTK-inoziva kuyera uye YaRN yakakwenenzvera nekududzira ma frequency zvakanyanya kusarudza.