Isandiso sobude be-YaRN kanye Nokuqukethwe
I-YaRN (Nokho enye isandiso se-RoPE) iyindlela ephumelelayo yokwelula iwindi lomongo elisebenzisekayo lemodeli lize lidlule lapho liqeqeshwe khona.
Uhlolojikelele
It cleverly rescales rotary position embeddings so a model trained on, say, 4K tokens can handle 32K or more with minimal fine-tuning.
I-Deep Dive
Ama-LLM amaningi esimanje ahlanganisa izindawo zamathokheni nge-RoPE (I-Rotary Position Embeddings), ezungezisa umbuzo nama-vector angukhiye ngama-engeli aboshelwe endaweni. Uma uphakela ukulandelana okude kunobude bokuqeqeshwa, lokhu kuzungezisa kungena kububanzi obungabonakali futhi imodeli iyaphuka. I-YaRN, eyethulwe ngo-2023 ngu-Bowen Peng nabahlanganyeli, ilungisa lokhu ngokutolikwa kwe-NTK-aware okusetshenziswa imvamisa ngayinye: ishiya izilinganiso zefrikhwensi ephezulu (ethwebula ubudlelwano bendawo, bebanga elifushane) ikakhulukazi ingakathintwa ngenkathi ihlanganisa ubukhulu befrikhwensi ephansi (okulandelela indawo yebanga elide). I-YaRN iphinde yengeze ukulungiswa kwezinga lokushisa ekunakekelweni ukuze kuliwe noshintsho lwe-entropy oluvela kuzimo ezinde. Umphumela uwukusebenza okuqinile kokuqukethwe okude ngemva kokulungiswa kahle engxenyeni encane yedatha nezinyathelo ezidingwa izindlela ezingenangqondo.
I-Technical Insight
I-RoPE inika ubukhulu bokushumeka ngakunye imvamisa yokuzungezisa. I-Naive linear interpolation icindezela wonke amafrikhwensi ngokulinganayo, ilimaza izilinganiso zefrikhwensi ephezulu ehlanganisa imininingwane yendawo. I-YaRN isebenzisa umsebenzi werempu ukuhlanganisa kuphela ubukhulu befrikhwensi ephansi (ubude begagasi ende) kuyilapho ilondoloza amafrikhwensi aphezulu, kanye nesikali sokushisa sokunaka esingu-1/sqrt(t) esigcina ukucija kwe-softmax kuzinzile njengoba ubude bokulandelana bukhula. Le ndlela ye-NTK ngezingxenye inweba umongo ngokuwohloka okuncane kakhulu.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Le-YaRN kanye Nesandiso Sobude Bengqikithi
Isandiso somongo manje sekuwumkhuba ojwayelekile: amamodeli avulekile ajwayele ukuthumela okuhlukile okunwetshiwe kwe-YaRN afinyelela kumathokheni angu-128K noma ngaphezulu. Ucwaningo luya ezindleleni ezinweba umongo ngo-zero noma eduze noziro ukulungisa kahle, ukuhlanganisa i-RoPE rescaling namaqhinga ephethini yokunaka, futhi igcine ikhwalithi kulo lonke iwindi eligcwele kuneziphetho nje. Lindela ukuhlanganiswa okuqinile kwalezi zindlela ekuqeqesheni kusengaphambili umongo omude kakhulu kunomdabu kunokuba ufakwe kabusha.
Ukuqaliswa Komhlaba Wangempela
Ukunweba imodeli evulekile ye-4K-context ibe ngu-32K noma 128K yombuzo wedokhumenti ende ephendulwa ngokulungisa kafushane
Ivumela amasistimu wokubuyisa-akhulisiwe ukuze angenise amaphaseji amaningi axhumene ngaphandle kokuncishiswa
Abasizi bekhodi yokunikeza amandla abadinga lonke ifayela eliyinqolobane elikhulu noma amafayela amaningi ngokushesha okukodwa
Ukujwayela imodeli eyisisekelo yezingxoxo ezinde zezingxoxo eziningi eziqongelela imilando emikhulu yengxoxo
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the YaRN and Context Length Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Position Interpolation for Context Extension
Imibuzo evame ukubuzwa
What is YaRN and Context Length Extension?
I-YaRN (Nokho enye isandiso se-RoPE) iyindlela ephumelelayo yokwelula iwindi lomongo elisebenzisekayo lemodeli lize lidlule lapho liqeqeshwe khona. Ngobuhlakani ikala ukushumeka kwezindawo ezijikelezayo ukuze imodeli eqeqeshelwe, ithi, amathokheni e-4K angakwazi ukuphatha u-32K noma ngaphezulu ngokulungiswa okuncane okuncane.
Iyiphi indlela yokubhala ikhodi yendawo elungiswa yi-YaRN ukuze inwebe umongo?
I-YaRN, igama layo elisho Esinye isandiso seRoPE, ikhulula ukushumeka kwendawo ejikelezayo esetshenziswa kuma-LLM amaningi esimanje.
Yini engahambi kahle uma imodeli ye-RoPE ibona ukulandelana okude kunobude bayo bokuqeqeshwa?
Izikhundla ezingaphezu kokuqeqeshwa zikhiqiza ama-engeli aphendukayo imodeli engakaze iwabone, ngakho ukuziphatha kokunaka kwehla kakhulu.
Ngabe i-YaRN iphatha kanjani izilinganiso zamafrikhwensi e-RoPE?
I-YaRN isebenzisa irempu ye-NTK-by-part ehlanganisa ukufiphala kwefrikhwensi ye-waveleng ende futhi ishiye ukufiphala kwe-high-frequency yebanga elifushane ngokuvamile kuphelele.
Ngaphandle kokukala kabusha amafrikhwensi, yikuphi ukulungisa okwengeziwe okusebenzayo kwe-YaRN?
I-YaRN yengeza ukukala kwezinga lokushisa kulogigi yokunaka ukuze imelane nokuguquguquka kwe-entropy okwenzeka njengoba umongo ukhula.
Iyiphi inzuzo ebalulekile esebenzayo ye-YaRN ngaphezu kokuhumusha okungaqondakali?
I-YaRN ifinyelela ikhwalithi eqinile yomongo omude ngemva kokulungiswa kahle engxenyeni encane yezindlela ezingenalwazi zedatha ezidingekayo.