Interpolaasioŋu posisioŋ ngir yokk ëmbiit li
Interpolation Position (PI) xarala la buy tàllal palanteeru contexte modelu làkk bi fu sori bamu weesu guddaayu tàggat bi, ci di eskalewaat indices positionnel yi ci barabu leen extrapoler.
Résumé
It lets a model trained on, say, 2K or 4K tokens handle 32K or more with only light fine-tuning.
Plongeur bu xóot
LLM yi gëna bari ci jamono jii dañuy jëfandikoo ay position yuy wëréelu (RoPE), yuy kode position ni angle yu ñuy wëréelu yuñ jëfandikoo ci laaj ak vecteur key. Sooy dundal ay sequence yu gëna gudd, model bi dafay gis position ak angle rotation yi mu musul tàggat, ba noppi performance bi dafay daanu ndax bàyyi xel bi dafay extrapole bu baax ci frequence yi nekk ci biti. Interpolaasioŋ position dafay moytu extrapolation: ngir tàmbalee ci guddaay L dem ba ci guddaay L', dafay xaaj index position bu nekk ak facteur L'/L, daal di gëna xëcc rang bu bees bi ci diggante biñ tàggat. Modèle bi leegi du gis lenn ludul angle distribution, di gëna dëgër diggante bi. Fine-tune bu gàtt (dafay faral di nekk ay téemeeri wala junni jéego) daf koy may mu mëna ànd ak diggante bu gëna ndaw, ba noppi am jeffin ju yàgg ju dëgër ci njëgu tàggat yaram bu ndaw.
Gis-gis xarala
RoPE dafay wëlbati ñaari dimension ci fréquence yuy tàmbalee ci lu woyof ba ci lu rëy. PI dafay soppi position m ci m/s fu s = L'/L, suko defee angle rotation yi des ci biir rang biñ tàggat, duñu extrapoler. Variante yiy xam fréquence yu melni NTK-aware scaling ak YaRN dem nañu gëna sori: dañuy wàññi fréquence yu woyof yi, fréquence yu kawe yi di gëna bari (wala ñu interpole ci guddaayu onde bi), di denc detay yu fréquence yu kawe yi ci noonu lañuy yokk frequence yu woyof yi.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu interpolaasioŋ position ngir yokk muy wax
Extension contexte mingi gaaw. Pexe yu melni NTK-xam RoPE scaling, YaRN, ak dynamic / long-RoPE leegi dañuy puus palanteer yi ba téemeeri junni wala sax ay milioŋ ciy token, yenn saa mu am tuuti wala amul benn ajustement. Xaarandil ni pexe scaling yii dina ñu boole ak bàyyi xel bu baax ak KV-cache compression, ñu nekk knob yuñ miin ci configs model yi. Gëstu bi dafay wéy ci tëye njubte gi ci palanteer bi yépp suko defee contexte yu gudd yi mën nañu ko jëfandikoo dëgg, duñu ko jàppale ci tur rek.
Doxal ci àdduna dëgg
Yaatalal xeetu LLaMA buñu tàggat ci 4K ba ci muy tënk 32K ngir tënk dokimaa yu gudd ginaaw biñu defee ay fine-tuning yu gàtt.
Yab codebase lëmm wala kontraa legal bu mag ci benn laaj ngir tontu laaj cross-file.
Jëfandikoo NTK-aware wala YaRN scaling ngir yokk contexte bi ak tuuti wala amul benn tàggat.
Joxe ay jaar-jaari waxtaan yu gudd te duñu dagg, jaaraleko ci eskalaasioŋ bu bees ci RoPE ci waxtu inference.
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Position Interpolation for Context Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
YaRN ak guddaayu ëmbiit li
Laaj yi ñuy faral di laaj
What is Position Interpolation for Context Extension?
Interpolation Position (PI) xarala la buy tàllal palanteeru contexte modelu làkk bi fu sori bamu weesu guddaayu tàggat bi, ci di eskalewaat indices positionnel yi ci barabu leen extrapoler. Dafay may benn xeetu tàggat ci, wax, 2K wala 4K tokens jëfandikoo 32K wala lu ëpp ak fine-tuning bu woyof.
Lan mooy xalaat bu mag bi ci interpolaasioŋ position?
PI dafay xaaj indices position yi ak facteur extension bi, mu delloosi sequence bi gëna gudd ci biir distribution bi model bi jàngoon ba pare.
Ban xeetu enkodaasu posisioŋ moo gëna méngoo ak interpolaasioŋ posisioŋ?
PI dañu ko siiwal ci xeetu RoPE, di soppi wàll yi ñuy wëréelu ngir ñu des ci fréquence yiñ tàggat.
Lan moo waral extrapolation naïf ci contexte yu gëna gudd di faral di lajj?
Feeding sequences yu gëna gudd dafay wane model bi ci angle yu sori yu mu musul tàggat, moo waral bàyyi xel ak performance di wàññeeku bu baax.
Sudee dañuy yokk guddaayi contexte ci L ba ci L', ban facteur de rééchelle la PI bu njëkk bi di jëfandikoo ci position m?
PI dafay wane m ba m xaaj ko ak facteur s = L'/L, di tënk rang bi gëna gudd ci diggante biñ njëkka tàggat.
naka lay def ba NTK-xam scaling ak YaRN di yokk ci interpolaasioŋ position bu leer?
Pexe yooyu dañuy wuute ci eskaalu fréquence yu woyof ak yu kawe, ñuy tëye ay detay yu ndaw ci barab bi, fekk ñuy yegg ci contexte yu gëna gudd.