interpolaasioŋ posisioŋ ngir muy lu gudd
Interpolation positionnelle (PI) xarala bu yomb la, am doole buy yokk palanteeru contexte Transformateur ci di tas indices position yu bees ci biir rang bi model bi xamoon ba pare.
Résumé
Instead of extrapolating to unseen positions, it interpolates within trained ones, requiring only brief fine-tuning.
Plongeur bu xóot
Meta gëstukat yi (Chen et al.) ñoo ko dugal ci 2023, muy wax ci ni model yi am RoPE dañuy lajj bu baax suñu leen jëlee ci position yu weesu tàggat. Gis-gis bi amul benn werante: ludul laaj model bi mu jëfandikoo valeur position yu gëna mag yu mu musul gis, PI dafay xaaj indices position yiy dugg ci facteur scale suko defee guddaay biñ bëgga wax, wax, 8K maps dellu ci 2K bi njëkk. Ndax model bi dañu ko tàggat ci rang boobu, rotation yi dañuy des ci distribution bi. Ginaaw 1000 jéego yu baax, benn xeetu LLaMA dafa yokk ci anam wii ñu mëna jëfandikoo ba ci 32K context. Këyit dafa wane ni extrapolation mën na yokk poñ yi ñuy bàyyi xel ci valeur yu bari, waaye interpolation daf leen di tëye, moo tax interpolation di gëna mëna liggéey extrapolation.
Gis-gis xarala
PI dafay soppi position m ci m/s fu s nekk facteur extension (lu melni, guddaay bu bees xaaj guddaay bu njëkk bi). Ci RoPE loolu dafay wàññi bu baax jéego yi ci wëréelu diggante position yi ci wetam, muy gëna yokk position yi ci rang angular biñ tàggat. Teori biñu tënk ci këyit bi dafay wane ni poñ yiñ joxe ci bàyyi xel dañuy des ci anam wu baax, waaye extrapolation naïf mën na defar poñ yu gëna mag ci lépp luñu gis ci tàggat yaram, di destabiliser softmax.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu interpolaasioŋ posisioŋ ngir muy lu yàgg
Interpolation positionnelle moo nekkoon fundamaasu topp-topp yu bari, lu ci melni NTK-aware scaling ak YaRN, ñoom ñooy gëna tànneef ngir baña yàq ay detay yu bari. Trajectoire bi mingi jubal ci pexe yu soxla tuuti wala amul benn fine-tuning ak jubal ci jëfandikoo contexte bu yàgg ci pretraining. PI mingi wéy di nekk royuwaay bu am solo te dañu koy faral di boole ak ay pexe yu bees yu xam fréquence ngir yegg ci palanteer yu 128K-plus ci anam wu jaar yoon.
Doxal ci àdduna dëgg
Yaatalal ab xeetu LLaMA bu 2K ngir mëna jëfandikoo ay jeton 8K-32K ak luy tollu ci 1000 jéego
Defar ab xeetu waxtaan bu fi nekk ngir tënk ay këyit yu gudd te doo tàggataat ci noonu
Di nekk royuwaay bi NTK-aware scaling ak YaRN di gëna suqali
Aktiwise kodu kontekst bu gudd wala jàngat këyitu yoon ci model yiñ njëkka tàggat ci palanteer yu gàtt
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Positional Interpolation for Long Context quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Interpolaasioŋu posisioŋ ngir yokk ëmbiit li
Laaj yi ñuy faral di laaj
What is Positional Interpolation for Long Context?
Interpolation positionnelle (PI) xarala bu yomb la, am doole buy yokk palanteeru contexte Transformateur ci di tas indices position yu bees ci biir rang bi model bi xamoon ba pare. Du extrapoler ci position yuñu gisul, waaye dafay interpoler ci biir position yuñ tàggat, te soxlawul ludul ay fine-tuning yu gàtt.
Lan mooy xalaat bu mag bi ci ginaaw interpolaasioŋ positionel?
PI dafay xaaj indices position yi ci facteur de scale suko defee ñu mëna dellu ci position yiñ tàggat, ba noppi di tëye rotation yi ci distribution bi.
Lu tax extrapolation bu yomb ci position yu gëna gudd du am njariñ?
Këyitu PI dafa wane ni position yu yaatu yuñu gisul mën nañu jur poñ yu gëna mag ci tàggat yaram, di destabiliser softmax.
Lu tollu ci ban fine-tuning la liggéeyu PI bi njëkk soxla ngir yokk LLaMA ba 32K?
Këyit dafa wax ni lu tollu ci 1000 jéego yu baax doyoon nañu ngir yokk muy wax ba 32K tokens.
Soo yokke contexte ci facteur s, naka la PI di soppi position m?
PI dafay wàññiwaat position m ba m/s, daal di tënk rang bu bees bi gëna mag ci rang biñ njëkka tàggat.
naka la PI amee njeexital ci yeneen xeeti xam-xam yu melni YaRN?
PI dafa taxawal interpolation muy anam wu wóor wi; NTK-xam scaling ak YaRN setal ko ci interpoler fréquences ci anam wu gëna tànneef.