YaRN ak guddaayu ëmbiit li
YaRN (Beneen RoPE extension) xarala bu am solo la ngir tàllal palanteer bi ñuy jëfandikoo ci model bi, dem fu sori ci li ñu ko tàggat.
Résumé
It cleverly rescales rotary position embeddings so a model trained on, say, 4K tokens can handle 32K or more with minimal fine-tuning.
Plongeur bu xóot
LLM yi gëna bari ci jamono jii dañuy kode position token yi ak RoPE (Embeddings Position Rotary), ñuy wëlbati laajte ak vecteur clé ci angle yuñ takk ci position bi. Soo feed sequence yu gëna gudd guddaayi tàggat yaram, rotation yooyu dañuy dugg ci rang yuñu gisul, model bi dafay yàqu. YaRN, bi Bowen Peng ak ay naataangoom dugal ci 2023, defar na lii ci interpolation NTK-aware buñu jëfandikoo ci fréquence bu nekk: dafay bàyyi dimension yu fréquence yu kawe (yiy jàpp diggante yu am diggante yu gàtt) yu bari te kenn laalu ko ci diggante dimension yu fréquence yu néew (yiy trapoler). YaRN itam dafay yokk tàngoor wiñ koy méngale ak coppite yi ci entropie yiy bawoo ci diggante yu gëna gudd. Resultaa bi mooy performance bu am doole ci contexte bu yàgg ginaaw biñu defaree ay done yu ndaw ak jéego yi approche naïf yi soxla.
Gis-gis xarala
RoPE dafay jox bépp dimension buñ samp fréquence buy wëréelu. Interpolaasioŋ ligneer bu yomb dafay tënk bépp fréquence ci anam wu tolloo, di yàq dimension fréquence yu kawe yiy kode ay detay yu baax ci gox bi. YaRN dafay jëfandikoo fonction rampe ngir interpoler ci dimension yu fréquence yu woyof yi (guddaayi onde yu gudd) yi, boole ci 1/sqrt (t) tàngoor wuy bàyyi xel ci scaling biy tëye softmax sharpness bi gëna mag. Bii NTK-by-parts gis-gis bi dafay yokk muy gëna néew luñuy yàq.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu YaRN ak yokk guddaayi contexte
Extension contexte leegi dafay nekk jëf buñ miin: model yu ubbeeku yi dañuy faral di yónnee YaRN-extended variants yuy yegg ba 128K tokens wala lu ëpp loolu. Gëstu mingi jëm ci pexe yuy yokk muy wax ak zero wala jege-zero fine-tuning, boole RoPE rescaling ak tricks motif attention, ak mën topp kalite ci palanteer bi yépp te baña yam ci njeexte yi. Xaarandi boole bu gëna dëgër ci pexe yii ci tàggat-yaram bu yàgg bi nekk native moo gën retrofitted.
Doxal ci àdduna dëgg
Yaatalal ab xeetu 4K-context bu ubbeeku ba 32K wala 128K ngir tontu laaj yu guddu ci këyit ak ab ajustement bu gàtt
May sistem yuñ yokk ngir seetlu ngir mëna jël paas yu bari yuñ boole te duñu dagg
Kodu assistant yi soxla fichier depositoire bu yaatu wala fichier yu bari ci benn prompt
Defar ab xeetu base ngir waxtaan yu gudd yu bari tur yuy dajale jaar-jaari waxtaan yu bari
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the YaRN and Context Length Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Interpolaasioŋu posisioŋ ngir yokk ëmbiit li
Laaj yi ñuy faral di laaj
What is YaRN and Context Length Extension?
YaRN (Beneen RoPE extension) xarala bu am solo la ngir tàllal palanteer bi ñuy jëfandikoo ci model bi, dem fu sori ci li ñu ko tàggat. Dafay soppi position rotary ci anam wu xarañ, suko defee benn model buñu tàggat ci, wax, jetons 4K mën na jëfandikoo 32K wala lu ëpp ak fine-tuning bu néew.
Ban anam enkodaasu position la YaRN di soppi ngir yokk muy wax?
YaRN, bi turam tekki beneen RoPE extension, dafay soppi position rotary bi ñuy jëfandikoo ci LLM yu bees yi.
Lan mooy jaarul yoon sudee modelu RoPE gisna ay toppalante yu gëna gudd guddaayi tàggat yaram?
Position yi weesu tàggat yaram dañuy defar angle rotation yu model bi musul gis, moo tax bàyyi xel ci doxalin bi dafay wàññeeku bu baax.
naka la YaRN di def ay dimension fréquence RoPE yu wuute?
YaRN dafay jëfandikoo rampe NTK-by-parts buy boole ay dim yu am fréquence yu woyof yu am guddaayu onde yu gudd, ba noppi bàyyi ay dim yu am fréquence yu kawe yu gàtt yu amul benn sikk.
Lu weesu wàññiwaat fréquence yi, ban coppite la YaRN di def?
YaRN dafay yokk tàngoor ci logits yi ngir jàpp ci coppite entropie yiy am su contexte bi di màgg.
Lan mooy njariñu YaRN ci kaw interpolation bu xamul dara?
YaRN defna kalite bu dëgër ci contexte bi ginaaw biñu ko fine-tuning ci paccu done yu ndaw yi pexe yu naïf yi soxla.