Ikirangantego cya YaRN
YaRN (Nyamara ubundi RoPE extensioN) ni tekinike irambura idirishya rikoreshwa rya transformateur irenze kure ibyo yatojwe, hamwe na tuning-nziza.
Incamake
It matters because it lets existing models handle much longer documents without retraining from scratch.
Kwibira cyane
LLMs nyinshi zigezweho zifata imyanya ikoresheje Rotary Position Embeddings (RoPE), ikora neza kugeza muburebure icyitegererezo cyabonye mugihe cy'amahugurwa. Kugaburira murwego rurerure kandi moderi itesha agaciro nabi. YaRN irabikemura mugukuraho imirongo ya RoPE muburyo bwo kumenya inshuro nyinshi: ibipimo byinshyi nyinshi (bifata umubano waho, hafi yubusabane) bisigara ahanini bidakorwaho, mugihe ibipimo bito bito (bifata umwanya muremure) birahujwe. Yongeraho kandi ubushyuhe bwo guhindura kugirango witondere kugirango logit yitware neza kurwego rurerure. Igisubizo, cyerekanwe kuri moderi ya LLaMA, cyagura imiterere kuva kuri 4K kugeza kuri 64K-128K ibimenyetso ukoresheje hafi 0.1% gusa yamakuru yambere yimyitozo nintambwe magana yo gutunganya neza.
Ubushishozi
RoPE izunguruka ikibazo hamwe nurufunguzo rwibanze ku mpande zingana n'umwanya hamwe na buri murongo. Naive umurongo interpolation (Umwanya Interpolation) uhuza imirongo yose uko ingana, byangiza amakuru yaho. YaRN ahubwo ikoresha 'NTK-by-ibice': ihuza gusa ibipimo buke-buke (uburebure-burebure), igasiga imirongo myinshi yonyine, kandi ikagenda hagati yabo. Igipimo cy'ubushyuhe cyitondewe cyishyura entropy ihinduka, ikabika neza uburebure burebure.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza ka YaRN Imirongo Idirishya
YaRN-yuburyo bwa frequence-izi kwaguka byahindutse ikintu cyambere cyo kohereza ibintu birebire; ibitandukanye nabasimbuye bakomeza kugaragara nkuko laboratoire isunika kuri miriyoni-token ya Windows. Tegereza kwishyira hamwe gukomeye hamwe no kwitabwaho neza, KV-cache compression, hamwe nogupima imbaraga bigenda bihinduka kuguruka kubisabwa. Inzira nini ni ugusibanganya 'igihe icyitegererezo cyatojwe' kuva 'igihe gishobora gusoma neza,' bigatuma imiterere ndende ihendutse nyuma yimyitozo aho kuba ubwubatsi buhenze.
Gushyira mu bikorwa Isi
Kwagura moderi ya LLaMA ifunguye kuva 4K kugeza 128K ibimenyetso kugirango ibashe kwinjiza codebase yose cyangwa amasezerano maremare muri pass imwe
Kureka chatbot ikagumana amateka maremare y'ibiganiro utagabanije guhinduka mbere
Incamake y'ibitabo birebire by'ibitabo cyangwa inyandiko-mvugo y'amasaha menshi arenga icyitegererezo cy'ibanze kavukire
Guhindura bihendutse moderi yabanje gutozwa kubikorwa birebire-byo kugarura ibintu ukoresheje akantu gato keza neza
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the YaRN Context Window Scaling quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Context Windows
Ibibazo bikunze kubazwa
What is YaRN Context Window Scaling?
YaRN (Nyamara ubundi RoPE extensioN) ni tekinike irambura idirishya rikoreshwa rya transformateur irenze kure ibyo yatojwe, hamwe na tuning-nziza. Nibyiza kuko ireka moderi zihari zikoresha inyandiko ndende cyane zitarinze kwitoza.
Nuwuhe mwanya-kodegisi ya YaRN ihindura kugirango yongere uburebure?
YaRN isobanura 'Nyamara ubundi RoPE extensioN' kandi ikora mugukuraho imirongo yizunguruka ikoreshwa muri Rotary Position Embeddings.
Nigute YaRN ifata inshuro nyinshi nubunini buke bwa RoPE?
Uburyo bwa YaRN 'NTK-by-ibice' burinda ibipimo byinshi (byaho) mugihe bihuza imirongo mike (ndende-ndende) kugirango birinde kwangiza amakuru yaho.
Ni izihe nyungu zingenzi za YaRN kurenza amahugurwa yerekana imiterere-ndende kuva kera?
YaRN irashobora kwagura imiterere ikoresheje hafi 0.1% yamakuru yambere yimyitozo hamwe numubare muto wintambwe nziza, ihendutse cyane kuruta imyitozo.
Usibye gukuraho inshuro nyinshi, ni ubuhe buryo bwiyongereye YaRN ikoresha kugirango ibitekerezo birebire bihamye?
YaRN ihindura ubushyuhe bwo kwitondera kugirango yishyure entropy / logit ihinduka ibaho mugihe ikurikiranye rirerire cyane.
Ni ikihe kibazo kibaho uramutse ugaburiye icyitegererezo cya RoPE igihe kirekire kuruta uko cyatojwe, nta gupima?
RoPE isanzwe muri rusange kugirango itagaragara imyanya ndende, bityo ubuziranenge burasenyuka keretse iyo imirongo yakuweho.