I-YaRN Context Window Scaling
I-YaRN (Nokho enye isandiso se-RoPE) iwubuchule obunweba iwindi lomongo elisebenzisekayo le-transformer kude kakhulu kunalokho ebiqeqeshelwa kukho, ngokulungiswa okuncane okuncane.
Uhlolojikelele
It matters because it lets existing models handle much longer documents without retraining from scratch.
I-Deep Dive
Ama-LLM amaningi esimanje ahlanganisa izindawo zamagama kusetshenziswa i-Rotary Position Embeddings (RoPE), esebenza kahle kuphela kuze kufike kubude obubonwe yimodeli ngesikhathi sokuqeqeshwa. Okuphakelayo ngokulandelana okude futhi imodeli yonakala kabi. I-YaRN ixazulula lokhu ngokukala kabusha amafrikhwensi e-RoPE ngendlela eqaphela imvamisa: izilinganiso zefrikhwensi ephezulu (ezithwebula ubudlelwano bendawo, obuseduze) zishiywa zingakathintwa kakhulu, kuyilapho izilinganiso zefrikhwensi ephansi (ezithwebula indawo yebanga elide) ziyahlanganiswa. Futhi yengeza ukulungiswa kwezinga lokushisa ekunakeni ukuze kugcinwe amalogi eziphatha kahle kumabanga amade. Umphumela, oboniswe kumamodeli we-LLaMA, unweba umongo ukusuka ku-4K ukuya kumathokheni angu-64K-128K kusetshenziswa kuphela cishe u-0.1% wedatha yokuqeqeshwa yasekuqaleni kanye nezinyathelo ezingamakhulu ambalwa zokulungisa kahle.
I-Technical Insight
I-RoPE izungezisa umbuzo namavekhtha angukhiye nge-engeli elinganayo nendawo kanye nefrikhwensi yobukhulu bobukhulu. I-Naive linear interpolation (Position Interpolation) icindezela wonke amafrikhwensi ngokulinganayo, ilimaza imininingwane yendawo. I-YaRN esikhundleni salokho isebenzisa i-'NTK-by-parts': ihlanganisa kuphela ubukhulu befrikhwensi ephansi (ubude begagasi elide), ishiya amaza aphezulu wodwa, kanye namarampu phakathi kwawo. Ukukalwa kwezinga lokushisa lokunaka kunxephezela ukushintsha kwe-entropy, okugcina ukunemba ngobude obunwetshiwe.
I-Strategic Impact
Isivinini nesikali
Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.
Finyelela futhi ufinyelele
Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.
Izinqumo ezicacile
Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.
Ikusasa Le-YaRN Context Window Scalling
Isandiso sokuqaphela imvamisa yesitayela se-YaRN sesiphenduke isithako esizenzakalelayo sokuthumela amamodeli anomongo omude; okuhlukile nabalandelayo baqhubeka nokuvela njengoba amalebhu ephushela efasiteleni lamathokheni ayisigidi. Lindela ukuhlanganiswa okuqinile ngokunaka okusebenzayo, ukucindezelwa kwenqolobane ye-KV, nokukala okuguquguqukayo okulungisa ekundizeni ngokwesicelo ngasinye. Ukuthambekela okubanzi ukuhlukanisa 'imodeli yaqeqeshwa isikhathi esingakanani' ukusuka kokuthi 'ingakwazi ukufunda isikhathi esingakanani,' okwenza umongo omude ube isici esishibhile sangemva kokuqeqeshwa esikhundleni sokuzibophezela kwezakhiwo ezibizayo.
Ukuqaliswa Komhlaba Wangempela
Ukunweba imodeli evulekile ye-LLaMA ukusuka ku-4K kuye kumathokheni angu-128K ukuze ingenise yonke i-codebase noma inkontileka ende ngephasi eyodwa.
Ukuvumela i-chatbot ukuthi igcine imilando emide kakhulu yezingxoxo ngaphandle kokunciphisa izikhathi zangaphambili
Ifinyeza amadokhumenti obude bencwadi noma imibhalo yamahora amaningi eyeqa iwindi lomdabu lemodeli eyisisekelo
Ukushintsha kalula imodeli eqeqeshwe kusengaphambili yemisebenzi yokubuyisa okuqukethwe okude kusetshenziswa ukuhlelwa okuncane kuphela
Izingozi & Guardrails
Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.
Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.
Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.
Ukuqalisa Umhlahlandlela
Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.
Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.
Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.
Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the YaRN Context Window Scaling quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
IWindows yokuqukethwe
Imibuzo evame ukubuzwa
What is YaRN Context Window Scaling?
I-YaRN (Nokho enye isandiso se-RoPE) iwubuchule obunweba iwindi lomongo elisebenzisekayo le-transformer kude kakhulu kunalokho ebiqeqeshelwa kukho, ngokulungiswa okuncane okuncane. Ibalulekile ngoba ivumela amamodeli akhona ukuthi aphathe amadokhumenti amade ngaphandle kokuphinda aqeqeshwe kusukela ekuqaleni.
Iluphi uhlelo lombhalo wekhodi we-YaRN olushintshayo ukuze lunwebe ubude bomongo?
I-YaRN imele okuthi 'Yet another RoPE extensionN' futhi isebenza ngokukhulula amaza okuzungezisa asetshenziswa Ekushumekelweni Kwesimo Se-Rotary.
I-YaRN iyiphatha kanjani i-high-frequency uma iqhathaniswa ne-low-frequency ye-RoPE?
Indlela ye-YaRN's 'NTK-by-parts' igcina ubukhulu bemvamisa ephezulu (yasendaweni) ngenkathi ihlobanisa imvamisa ephansi (ububanzi obude) ukugwema ukulimaza imininingwane yendawo.
Iyiphi inzuzo eyinhloko yokusebenza kahle kwe-YaRN ngaphezu kokuqeqesha imodeli yombhalo omude kusukela ekuqaleni?
I-YaRN inganweba umongo isebenzisa cishe u-0.1% wedatha yokuqeqeshwa yasekuqaleni kanye nenani elincane lezinyathelo zokulungisa kahle, ishibhe kakhulu kunokuphinda uziqeqeshe.
Ngaphandle kokukala kabusha amafrikhwensi, yikuphi ukulungisa okwengeziwe i-YaRN esebenzayo ukuze kugcinwe ukunaka kwebanga elide kuzinzile?
I-YaRN ilungisa izinga lokushisa lokunaka ukuze linxephezele ukushintsha kwe-entropy/logit okwenzeka lapho ukulandelana kuba kunde kakhulu.
Iyiphi inkinga eyenzeka uma uphakela imodeli ye-RoPE ukulandelana kwayo isikhathi eside kunokuqeqeshwa kwayo, ngaphandle kokukalwa?
I-RoPE ijwayele ukujwayela kabi ezindaweni ezinde ezingabonakali, ngakho ikhwalithi iyehla ngaphandle uma amaza enziwe kabusha.