YaRN Palanteer Kontekst Eskalaasioŋ
YaRN (Beneen RoPE extension) xarala la buy tàllal palanteer bi transformateur bi mëna jëfandikoo fu sori lool ci li ñu ko tàggat, ak tuuti ay fine-tuning.
Résumé
It matters because it lets existing models handle much longer documents without retraining from scratch.
Plongeur bu xóot
LLM yi gëna bari ci jamono jii dañuy kode position baat yi ci jëfandikoo Rotary Position Embeddings (RoPE), te loolu dafay dox bu baax ba ci guddaay bi model bi gis ci diiru tàggat bi. Feed ci yoon wu gëna gudd te model bi dafay yàqu bu baax. YaRN dafay saafara jafe-jafe yii ci wàññiwaat fréquence wëréelu RoPE ci anam wu xam fréquence yi: dimension fréquence yu kawe yi (yiy jàpp relasioŋ yi ci gox bi, jege) dañu leen bàyyi ñu laal leen, waaye dimension yu fréquence yu néew yi (yiy jàpp position yu sori) dañu leen di boole. Dafay yokk itam benn yamale tàngoor ngir bàyyi xel ci logit yi ñu mëna doxal bu baax ci diggante yu sori. Resultaa bi, ñu wane ko ci xeetu LLaMA, dafay yokk contexte ci 4K ba 64K-128K tokens jëfandikoo lu tollu ci 0.1% ci done tàggat yu njëkk yi ak ay téemeeri jéego yu baax.
Gis-gis xarala
RoPE dafay wëlbati vecteur laaj ak caabi ci angle bu méngoo ak position bi ak fréquence bu dimension bu nekk. Interpolaasioŋ ligneer bu yomb (interpolaasioŋ posisioŋ) dafay yàq bépp fréquence buy tolloo, di yàq detay yi ci gox bi. YaRN lu moy loolu dafay jëfandikoo 'NTK-by-parts': dafay boole dimension yu fréquence yu woyof yi (guddaayi onde yu gudd), bàyyi yu fréquence yu kawe yi kese, ak ay rampe ci seen biir. Eskalaasioŋu tàngoor wiñ bàyyi xel mooy kompensaasioŋ coppite entropi bi, ba noppi di tëye njub ci guddaay yu gëna gudd.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu YaRN Kontekst Scaling Palanteer
Extension biy xam fréquence bu nuru YaRN nekk na ingredient biñ jagleel ngir yobbu ay model yu am contexte bu gudd; variants ak ñiy wuutu ñu ngi wéy di feeñ bi labs yi di puus ci palanteer yu am milioŋ yu token. Xaarandil lëkkaloo bu gëna dëgër ak bàyyi xel bu jaar yoon, KV-cache compression, ak scaling dynamique buy méngoo ci laaj bu nekk. Tendens bi gëna yaatu mooy dindi 'ba ñaata fan lañu tàggat benn model' ak 'ba ñaata fan lay mëna jàng bu baax,' def contexte bu gudd muy màndarga bu yomb ginaaw tàggat bi moo gëna nekk architecture bu seer.
Doxal ci àdduna dëgg
Yaatalal xeetu LLaMA bu ubbeeku ci 4K ba 128K jetons suko defee mu mëna jël codebase bi yépp wala kontraa bu gudd ci benn yoon
Bayyi ab chatbot mu denc ay jaar-jaari waxtaan yu gudd te du dagg turn yu njëkk yi
Résumé dokimaa yu am guddaayu téere wala transkripsioŋ yu waxtu yu bari yu weesu palanteer biñ defaree modelu base bi
Defar ci anam wu yomb ab model buñu tàggat bu njëkk ngir ay liggéey yu gudd ngir seet ay mbir yu jëfandikoo ab daw bu ndaw
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the YaRN Context Window Scaling quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Windows Kontekst
Laaj yi ñuy faral di laaj
What is YaRN Context Window Scaling?
YaRN (Beneen RoPE extension) xarala la buy tàllal palanteer bi transformateur bi mëna jëfandikoo fu sori lool ci li ñu ko tàggat, ak tuuti ay fine-tuning. Dafa am solo ndax dafay may model yi fi nekk ñu mëna jëfandikoo dokimaa yu gëna gudd te duñu leen tàggataat ci noonu rek.
Ban xeetu kodage position la YaRN di soppi ngir yokk guddaayi contexte bi?
YaRN mooy 'Beneen yokk bu RoPE' te dafay liggéey ci wàññiwaat fréquence yi ñuy jëfandikoo ci Embeddings Position Rotary.
naka la YaRN di doxalee ci dimension RoPE yu fréquence yu kawe ak yu fréquence yu néew?
YaRN's 'NTK-by-parts' dafay baña yàq dimension yu fréquence yu kawe yi (ci gox bi) ci noonu lañuy boole yu fréquence yu néew yi (distance yu sori) ngir moytu yàq detay yi ci gox bi.
Lan mooy njariñu efficacité bu mag bi YaRN am ci tàggat model bu yàgg ci noonu?
YaRN mën na yokk muy wax ci jëfandikoo lu tollu ci 0.1% ci done yiñ njëkka amal ak ay jéego yu néew yuñ defar, te loolu moo gëna xéewale tàggataat.
Lu weesu wàññiwaat frequency yi, ban coppite la YaRN di def ngir mëna wéy di bàyyi xel ci fu sori?
YaRN dafay soppi tàngooru bàyyi xel ngir kompensaasioŋ entropi / logit shift biy am su toppalante yi gëna gudd.
Ban jafe-jafe mooy xew sudee dangay dundal benn model RoPE ci anam wu gëna gudd ci li ñu ko tàggat, te amul benn scaling?
RoPE dafay generalise bu baax ci position yu gudd yuñu gisul, moo tax kalite bi dafay daanu fileek frequency yi deñu leen escalewul.