InoteveraGaidhi rinotevera
Position Interpolation yeContext Extension
Mutauro AI
Nhungamiro yehunyanzvi
YaRN uye Context Length Kuwedzerwa
YaRN (Zvakadaro imwezve RoPE yekuwedzeraN) inzira inoshanda yekutambanudza modhi inoshandisika hwindo hwindo kure kupfuura yayakadzidziswa pairi.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Pa peji ino2 min verenga
Pfupiso
Iyo nehungwaru inodzoreredza inotenderera chinzvimbo embeddings saka modhi yakadzidziswa pa, toti, 4K tokens inogona kubata 32K kana kupfuura nekushoma-tuning.
Kudzika Kwakadzika
Mazhinji maLLM emazuva ano anokodha zvinzvimbo zvetokeni neRoPE (Rotary Position Embeddings), iyo inotenderedza muvhunzo nemakiyi mavheta nemakona akasungirirwa pachinzvimbo. Kana iwe uchidyisa kutevedzana kwenguva yakareba kupfuura kureba kwekudzidziswa, kutenderera uku kunopinda zvisingaonekwe uye modhi inoputsika. YaRN, yakaunzwa muna 2023 naBowen Peng nevabatsiri, inogadzirisa izvi neNTK-inoziva kududzira yakashandiswa pafrequency: inosiya yakakwira-frequency dimensions (inobata hukama hwemunharaunda, hupfupi-renji) kazhinji isina kubatwa uku ichipindirana yakaderera-frequency zviyero (iyo inoteedzera-refu-renji chinzvimbo). YaRN inowedzerawo tembiricha inogadzirisa kutarisisa kurwisa shanduko ye entropy inouya kubva kurefu mamiriro. Mhedzisiro ndeyekuita kwakasimba-kwemamiriro ezvinhu mushure mekugadzirisa zvakanaka pachikamu chidiki che data nematanho anodiwa nenzira dzisina musoro.
Kuitwa muNyika yeChokwadi
Kuwedzera yakavhurika 4K-context modhi kusvika 32K kana 128K yemubvunzo-refu-gwaro mhinduro nekupfupika-tuning.
Kugonesa kudzoreredza-yakawedzera masisitimu kupinza akawanda akabatikana ndima pasina kuderedzwa.
Powering kodhi vabatsiri vanoda yakazara hombe repository faira kana akawanda mafaera mune imwe kukurumidza
Kugadzirisa muenzaniso wekutanga wehurukuro dzakareba dzakawanda dzinounganidza nhoroondo huru dzekutaura
Ramangwana reYaRN uye Context Length Extension
Kuwedzeredzwa kwemamiriro ezvinhu ikozvino kwakajairika: mamodheru akavhurika anogara achitakura YaRN-yakawedzerwa akasiyana anosvika 128K tokens kana kupfuura. Tsvagiridzo iri kuenda kunzira dzinowedzera mamiriro ne zero kana pedyo-zero yakanaka-tuning, sanganisa RoPE kudzoreredza nekutarisisa-patani maitiro, uye chengetedza mhando pahwindo rakazara kwete kungogumira. Tarisira kubatanidzwa kwakasimba kweaya matekiniki mukuita pretraining yakarebesa mamiriro ekuzvarwa kwete kudzokororwa.
Kuzvishandisa
Nei zvichikosha
- Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
- Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
- Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Chenjera
- Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
- Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
- Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Matanho anotevera
- Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
- Benchmark pasi pechokwadi mutoro uye data mamiriro.
- Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
- Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Tsamba yemahara
Tora iyo yemazuva ese AI pfupiso
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Take the YaRN and Context Length Extension quiz
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Mibvunzo inowanzo bvunzwa
Chii chinonzi YaRN uye Context Length Extension?
YaRN (Zvakadaro imwezve RoPE yekuwedzeraN) inzira inoshanda yekutambanudza modhi inoshandisika hwindo hwindo kure kupfuura yayakadzidziswa pairi. Iyo nehungwaru inodzoreredza inotenderera chinzvimbo embeddings saka modhi yakadzidziswa pa, toti, 4K tokens inogona kubata 32K kana kupfuura nekushoma-tuning.
Ndeipi chinzvimbo-encoding nzira inogadziriswa neYaRN kuti iwedzere mamiriro?
YaRN, ine zita rinoreva Kuti Imwezve RoPE extentionN, inodzoreredza nzvimbo inotenderedzwa inomisikidzwa inoshandiswa mumaLLM mazhinji azvino.
Chii chinotadza kana modhi yeRoPE ikaona kutevedzana kwakareba kupfuura kureba kwayo kwekudzidziswa?
Zvinzvimbo zvinopfuura kudzidziswa zvinoburitsa makona ekutenderera iwo modhi haana kumboona, saka hunhu hwekutarisisa hunodzikisira zvakanyanya.
YaRN inobata sei akasiyana RoPE frequency zviyero?
YaRN inoshandisa NTK-ne-zvikamu rampu inopindirana kureba-wavelength yakaderera-frequency dims uye inosiya pfupi-refu-yepamusoro-frequency dims kazhinji isina kusimba.
Kunze kwekudzoreredza ma frequency, ndeipi imwe gadziriso inoshandiswa neYaRN?
YaRN inowedzera tembiricha yekuyera kune yekutarisisa marogi kurwisa entropy shanduko inoitika sezvo mamiriro anokura.
Ndechipi chakakosha chinoshanda mukana weYaRN pamusoro pekusaziva kududzira?
YaRN inowana yakasimba-yakareba-mamiriro emhando mushure mekugadzirisa zvakanaka pachikamu chidiki che data naive nzira dzinodiwa.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi