Umwanya wa Interpolation kumwanya muremure
Umwanya wa Interpolation (PI) nubuhanga bworoshye, bukomeye bwagura idirishya ryimiterere ya Transformer ukanda indangagaciro nshya mumurongo icyitegererezo kimaze kumenya.
Incamake
Instead of extrapolating to unseen positions, it interpolates within trained ones, requiring only brief fine-tuning.
Kwibira cyane
Yatangijwe na Meta abashakashatsi (Chen et al.) Mu 2023, Positional Interpolation ikemura ikibazo cyuko moderi hamwe na RoPE zananirana bikabije mugihe zoherejwe mumyanya irenze amahugurwa. Ubushishozi buvuguruzanya: aho gusaba icyitegererezo gukora indangagaciro nini itigeze ibona, PI igabanya ibipimo byinjira byinjira ku gipimo cyerekana igipimo cyerekana uburebure bwa, kuvuga, amakarita 8K asubira mu mwimerere wa 2K. Kuberako icyitegererezo cyatojwe kururwo rwego, kuzunguruka biguma-kugabura. Nyuma yintambwe 1.000 gusa yo gutunganya neza, moderi ya LLaMA yaguye ubu buryo ikemura kugeza 32K. Uru rupapuro rwerekanye ko extrapolation ishobora gutesha agaciro amanota manini, mugihe interpolation ituma imipaka kandi itajegajega, niyo mpamvu interpolation ikora neza cyane kuruta kurenganurwa.
Ubushishozi
PI ikuraho umwanya m kugeza m / s aho s ni ikintu cyo kwagura (urugero, uburebure bushya bugabanijwe nuburebure bwumwimerere). Kuri RoPE ibi bigabanya neza intambwe yo kuzenguruka hagati yimyanya yegeranye, gupakira imyanya myinshi murwego rwahuguwe. Inyigisho zifatanije nimpapuro zerekana amanota yitondewe amanota akomeza kugenzurwa neza, mugihe naive extrapolation ishobora gutanga amanota yubunini burenze ikintu cyose kigaragara mumahugurwa, guhungabanya softmax.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Kazoza ka Interpolation ya Position kumwanya muremure
Umwanya wa Interpolation wabaye ishingiro ryumurongo wo gukurikirana, harimo NTK-uzi gupima na YaRN, ihuza cyane guhitamo kugirango ibungabunge amakuru yaho. Inzira nyabagendwa yerekeza kuburyo bukenera bike cyangwa budahuza neza no guteka ibintu birebire bikoreshwa mukwitegura. PI ikomeza kuba ibyingenzi kandi ikunze guhuzwa na gahunda nshya-yamenyeshejwe gahunda yo kugera kuri 128K-wongeyeho imiterere ya Windows neza.
Gushyira mu bikorwa Isi
Kwagura moderi ya 2K-ya LLaMA kugirango ikore ibimenyetso 8K-32K hamwe nintambwe zigera ku 1.000
Guhuza uburyo bwo kuganira buriho kubwinyandiko ndende incamake utarinze kwitoza
Gukora nkibisobanuro fatizo NTK-izi gupima na YaRN itezimbere
Gushoboza imirongo miremire cyangwa amategeko-inyandiko yisesengura kuri moderi yabanje gutozwa na Windows ngufi
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Positional Interpolation for Long Context quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Umwanya Interpolation yo Kwagura Imirongo
Ibibazo bikunze kubazwa
What is Positional Interpolation for Long Context?
Umwanya wa Interpolation (PI) nubuhanga bworoshye, bukomeye bwagura idirishya ryimiterere ya Transformer ukanda indangagaciro nshya mumurongo icyitegererezo kimaze kumenya. Aho gutondekanya imyanya itagaragara, irahuza hagati yabatojwe, bisaba gusa guhuza neza.
Ni ikihe gitekerezo cyibanze inyuma ya Interpolation?
PI igabanya ibipimo byerekana umwanya munini kuburyo ikarita ndende ikurikirana ikagaruka mumyitozo yatojwe, igakomeza kuzenguruka mukwirakwiza.
Ni ukubera iki naive extrapolation kumwanya muremure byananirana?
Urupapuro rwa PI rwerekanye imyanya minini itagaragara irashobora gutanga amanota yo gutondekanya ubunini burenze amahugurwa, guhungabanya softmax.
Nkanswe kangahe-gutunganya neza akazi ka mbere PI yasabye kwagura LLaMA kugeza 32K?
Uru rupapuro rwatangaje ko intambwe zigera ku 1.000 zo gutunganya neza zari zihagije kugira ngo ibintu bigere kuri 32K.
Niba waguye imiterere kubintu s, nigute PI ihindura umwanya m?
PI ikuraho umwanya m kugeza m / s, ikanda intera nini nini murwego rwambere rwahuguwe.
Nigute PI yagize ingaruka muburyo bwanyuma nka YaRN?
PI yashyizeho interpolation nkuburyo bwizewe; NTK-izi gupima na YaRN yarayinonosoye muguhuza imirongo inshuro nyinshi.