YaRN uye Context Length Kuwedzerwa
YaRN (Zvakadaro imwezve RoPE yekuwedzeraN) inzira inoshanda yekutambanudza modhi inoshandisika hwindo hwindo kure kupfuura yayakadzidziswa pairi.
Pfupiso
It cleverly rescales rotary position embeddings so a model trained on, say, 4K tokens can handle 32K or more with minimal fine-tuning.
Kudzika Kwakadzika
Mazhinji maLLM emazuva ano anokodha zvinzvimbo zvetokeni neRoPE (Rotary Position Embeddings), iyo inotenderedza muvhunzo nemakiyi mavheta nemakona akasungirirwa pachinzvimbo. Kana iwe uchidyisa kutevedzana kwenguva yakareba kupfuura kureba kwekudzidziswa, kutenderera uku kunopinda zvisingaonekwe uye modhi inoputsika. YaRN, yakaunzwa muna 2023 naBowen Peng nevabatsiri, inogadzirisa izvi neNTK-inoziva kududzira yakashandiswa pafrequency: inosiya yakakwira-frequency dimensions (inobata hukama hwemunharaunda, hupfupi-renji) kazhinji isina kubatwa uku ichipindirana yakaderera-frequency zviyero (iyo inoteedzera-refu-renji chinzvimbo). YaRN inowedzerawo tembiricha inogadzirisa kutarisisa kurwisa shanduko ye entropy inouya kubva kurefu mamiriro. Mhedzisiro ndeyekuita kwakasimba-kwemamiriro ezvinhu mushure mekugadzirisa zvakanaka pachikamu chidiki che data nematanho anodiwa nenzira dzisina musoro.
Technical Insight
RoPE inopa yega yega yekumisikidza dimension frequency yekutenderera. Naive linear kududzira inomanikidza ese mafrequency zvakaenzana, ichikuvadza hukuru-frequency zviyero izvo encode yakanaka yenzvimbo. YaRN inoshandisa ramp function kusanganisa chete yakaderera-frequency (yakareba-wavelength) uku ichichengetedza yakakwirira-frequency iwo, pamwe ne1/sqrt (t) yekutarisisa tembiricha kuyera iyo inochengetedza kupinza kweSoftmax kugadzikana sezvo kutevedzana kureba kuchikura. Iyi NTK-ne-zvikamu maitiro inowedzera mamiriro nekuderera kushoma.
Strategic Impact
Mutengo uye bhajeti
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
Sarudzo dzakajeka
Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
Kudzora kwemhando yepamusoro
Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Ramangwana reYaRN uye Context Length Extension
Kuwedzeredzwa kwemamiriro ezvinhu ikozvino kwakajairika: mamodheru akavhurika anogara achitakura YaRN-yakawedzerwa akasiyana anosvika 128K tokens kana kupfuura. Tsvagiridzo iri kuenda kunzira dzinowedzera mamiriro ne zero kana pedyo-zero yakanaka-tuning, sanganisa RoPE kudzoreredza nekutarisisa-patani maitiro, uye chengetedza mhando pahwindo rakazara kwete kungogumira. Tarisira kubatanidzwa kwakasimba kweaya matekiniki mukuita pretraining yakarebesa mamiriro ekuzvarwa kwete kudzokororwa.
Real-World Implementation
Kuwedzera yakavhurika 4K-context modhi kusvika 32K kana 128K yemubvunzo-refu-gwaro mhinduro nekupfupika-tuning.
Kugonesa kudzoreredza-yakawedzera masisitimu kupinza akawanda akabatikana ndima pasina kuderedzwa.
Powering kodhi vabatsiri vanoda yakazara hombe repository faira kana akawanda mafaera mune imwe kukurumidza
Kugadzirisa muenzaniso wekutanga wehurukuro dzakareba dzakawanda dzinounganidza nhoroondo huru dzekutaura
Njodzi & Guardrails
Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Implementation Roadmap
Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
Benchmark pasi pechokwadi mutoro uye data mamiriro.
Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the YaRN and Context Length Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Position Interpolation for Context Extension
Mibvunzo inowanzo bvunzwa
What is YaRN and Context Length Extension?
YaRN (Zvakadaro imwezve RoPE yekuwedzeraN) inzira inoshanda yekutambanudza modhi inoshandisika hwindo hwindo kure kupfuura yayakadzidziswa pairi. Iyo nehungwaru inodzoreredza inotenderera chinzvimbo embeddings saka modhi yakadzidziswa pa, toti, 4K tokens inogona kubata 32K kana kupfuura nekushoma-tuning.
Ndeipi chinzvimbo-encoding nzira inogadziriswa neYaRN kuti iwedzere mamiriro?
YaRN, ine zita rinoreva Kuti Imwezve RoPE extentionN, inodzoreredza nzvimbo inotenderedzwa inomisikidzwa inoshandiswa mumaLLM mazhinji azvino.
Chii chinotadza kana modhi yeRoPE ikaona kutevedzana kwakareba kupfuura kureba kwayo kwekudzidziswa?
Zvinzvimbo zvinopfuura kudzidziswa zvinoburitsa makona ekutenderera iwo modhi haana kumboona, saka hunhu hwekutarisisa hunodzikisira zvakanyanya.
YaRN inobata sei akasiyana RoPE frequency zviyero?
YaRN inoshandisa NTK-ne-zvikamu rampu inopindirana kureba-wavelength yakaderera-frequency dims uye inosiya pfupi-refu-yepamusoro-frequency dims kazhinji isina kusimba.
Kunze kwekudzoreredza ma frequency, ndeipi imwe gadziriso inoshandiswa neYaRN?
YaRN inowedzera tembiricha yekuyera kune yekutarisisa marogi kurwisa entropy shanduko inoitika sezvo mamiriro anokura.
Ndechipi chakakosha chinoshanda mukana weYaRN pamusoro pekusaziva kududzira?
YaRN inowana yakasimba-yakareba-mamiriro emhando mushure mekugadzirisa zvakanaka pachikamu chidiki che data naive nzira dzinodiwa.