Ci bi mu melGis bi ci topp
Interpolaasioŋu posisioŋ ngir yokk ëmbiit li
IA làkk
GUIDE teknik
YaRN ak guddaayu ëmbiit li
YaRN (Beneen RoPE extension) xarala bu am solo la ngir tàllal palanteer bi ñuy jëfandikoo ci model bi, dem fu sori ci li ñu ko tàggat.
Déglul tegtal biiIA-nettali · ~2 min
Ci xët wii2 simili jàng
Résumé
It cleverly rescales rotary position embeddings so a model trained on, say, 4K tokens can handle 32K or more with minimal fine-tuning.
Plongeur bu xóot
LLM yi gëna bari ci jamono jii dañuy kode position token yi ak RoPE (Embeddings Position Rotary), ñuy wëlbati laajte ak vecteur clé ci angle yuñ takk ci position bi. Soo feed sequence yu gëna gudd guddaayi tàggat yaram, rotation yooyu dañuy dugg ci rang yuñu gisul, model bi dafay yàqu. YaRN, bi Bowen Peng ak ay naataangoom dugal ci 2023, defar na lii ci interpolation NTK-aware buñu jëfandikoo ci fréquence bu nekk: dafay bàyyi dimension yu fréquence yu kawe (yiy jàpp diggante yu am diggante yu gàtt) yu bari te kenn laalu ko ci diggante dimension yu fréquence yu néew (yiy trapoler). YaRN itam dafay yokk tàngoor wiñ koy méngale ak coppite yi ci entropie yiy bawoo ci diggante yu gëna gudd. Resultaa bi mooy performance bu am doole ci contexte bu yàgg ginaaw biñu defaree ay done yu ndaw ak jéego yi approche naïf yi soxla.
Doxal ci àdduna dëgg
Yaatalal ab xeetu 4K-context bu ubbeeku ba 32K wala 128K ngir tontu laaj yu guddu ci këyit ak ab ajustement bu gàtt
May sistem yuñ yokk ngir seetlu ngir mëna jël paas yu bari yuñ boole te duñu dagg
Kodu assistant yi soxla fichier depositoire bu yaatu wala fichier yu bari ci benn prompt
Defar ab xeetu base ngir waxtaan yu gudd yu bari tur yuy dajale jaar-jaari waxtaan yu bari
Ëlëgu YaRN ak yokk guddaayi contexte
Extension contexte leegi dafay nekk jëf buñ miin: model yu ubbeeku yi dañuy faral di yónnee YaRN-extended variants yuy yegg ba 128K tokens wala lu ëpp loolu. Gëstu mingi jëm ci pexe yuy yokk muy wax ak zero wala jege-zero fine-tuning, boole RoPE rescaling ak tricks motif attention, ak mën topp kalite ci palanteer bi yépp te baña yam ci njeexte yi. Xaarandi boole bu gëna dëgër ci pexe yii ci tàggat-yaram bu yàgg bi nekk native moo gën retrofitted.
Teg ko ci jëf
Lu tax mu am solo
- Njëgg ak budget. Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
- dogal yu gëna leer. Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
- Xool kalite. Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Moytul
- Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
- Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
- Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Jéego yi ci topp
- Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
- Benchmark ci biir sargal ak done yu dëggu.
- Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
- Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Bulletin bu amul fayda
Jàngal xibaar AI bu bes
Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.
Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.
Nattal sa bopp
Take the YaRN and Context Length Extension quiz
Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.
Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax
Laaj yi ñuy faral di laaj
What is YaRN and Context Length Extension?
YaRN (Beneen RoPE extension) xarala bu am solo la ngir tàllal palanteer bi ñuy jëfandikoo ci model bi, dem fu sori ci li ñu ko tàggat. Dafay soppi position rotary ci anam wu xarañ, suko defee benn model buñu tàggat ci, wax, jetons 4K mën na jëfandikoo 32K wala lu ëpp ak fine-tuning bu néew.
Ban anam enkodaasu position la YaRN di soppi ngir yokk muy wax?
YaRN, bi turam tekki beneen RoPE extension, dafay soppi position rotary bi ñuy jëfandikoo ci LLM yu bees yi.
Lan mooy jaarul yoon sudee modelu RoPE gisna ay toppalante yu gëna gudd guddaayi tàggat yaram?
Position yi weesu tàggat yaram dañuy defar angle rotation yu model bi musul gis, moo tax bàyyi xel ci doxalin bi dafay wàññeeku bu baax.
naka la YaRN di def ay dimension fréquence RoPE yu wuute?
YaRN dafay jëfandikoo rampe NTK-by-parts buy boole ay dim yu am fréquence yu woyof yu am guddaayu onde yu gudd, ba noppi bàyyi ay dim yu am fréquence yu kawe yu gàtt yu amul benn sikk.
Lu weesu wàññiwaat fréquence yi, ban coppite la YaRN di def?
YaRN dafay yokk tàngoor ci logits yi ngir jàpp ci coppite entropie yiy am su contexte bi di màgg.
Lan mooy njariñu YaRN ci kaw interpolation bu xamul dara?
YaRN defna kalite bu dëgër ci contexte bi ginaaw biñu ko fine-tuning ci paccu done yu ndaw yi pexe yu naïf yi soxla.
Weyal di jàng
Gid yu jëm ci loolu
Tann nañu yeneen njiit ngir topic bii