Umwanya Interpolation yo Kwagura Imirongo
Umwanya Interpolation (PI) ni tekinike irambuye imiterere yururimi rwikoreshwa ryururimi rurenze kure uburebure bwamahugurwa mugukuraho indangagaciro zumwanya aho kuzikuramo.
Incamake
It lets a model trained on, say, 2K or 4K tokens handle 32K or more with only light fine-tuning.
Kwibira cyane
LLMs nyinshi zigezweho zikoresha kuzenguruka kumwanya (RoPE), zifata umwanya nkimpande zizunguruka zikoreshwa kubibazo nibice byingenzi. Niba ugaburira gusa urutonde rurerure, icyitegererezo kibona imyanya nu kuzenguruka bitigeze bigutoza, kandi imikorere irasenyuka kuko kwitondera gukabije kubitari kure yumurongo. Umwanya Interpolation irinda extrapolation: kwaguka kuva muburebure L kugeza kuburebure L ', igabanya buri cyerekezo cyerekana ibintu L' / L, ikanda intera nshya igasubira mumwanya watojwe. Icyitegererezo ubu cyonyine kibona-gukwirakwiza inguni, gusa gitandukanijwe cyane. Gitoya-tune (akenshi ni magana make kugeza ku gihumbi) ireka ikamenyera umwanya mwiza, itanga imyitwarire ihamye-ndende ku gipimo gito cyo kwitegura.
Ubushishozi
RoPE izunguruka ibipimo byombi kuri frequence igenda neza neza. PI ikuraho umwanya m kugeza kuri m / s aho s = L '/ L, bityo impande zizunguruka ziguma mumwanya wamenyerejwe aho gutandukana. Impinduka-zimenyekanisha inshuro nka NTK-zipima gupima na YaRN zijya kure: bapima imirongo mikeya nkeya kandi nini cyane (cyangwa interpolate kuburebure bwumuraba), ikabika umurongo-mwinshi waho mugihe cyo kwagura intera ndende-ndende.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Ejo hazaza h'imyanya Interpolation yo kwagura imiterere
Kwagura imiterere bigenda byihuta. Uburyo nka NTK izi RoPE gupima, YaRN, hamwe na dinamike / ndende-RoPE ubu isunika Windows kubihumbi magana cyangwa ndetse na miriyoni yibimenyetso, rimwe na rimwe hamwe na bike cyangwa bidafite neza. Tegereza aya mayeri yo gupima guhuzwa no kwitabwaho neza hamwe no kwikuramo KV-cache, no guhinduka ipfundo risanzwe muburyo bwimiterere. Ubushakashatsi burakomeza kugumya kumenya neza hejuru yidirishya ryuzuye kuburyo imirongo miremire ikoreshwa rwose, ntabwo ishyigikiwe gusa.
Gushyira mu bikorwa Isi
Kwagura moderi ya 4K yatojwe na LLaMA kumurongo wa 32K kugirango tuvuge muri make inyandiko ndende nyuma yo guhuza neza.
Gupakira codebase yose cyangwa amasezerano manini yemewe muburyo bumwe kugirango ikibazo cyambukiranya dosiye.
Gukoresha NTK-ibizi cyangwa YaRN gupima kugirango urambure imiterere hamwe na minimal cyangwa ntamahugurwa yinyongera.
Gukora amateka maremare yo kuganira utagabanije ukuraho imyanya ya RoPE mugihe cyo gufata umwanzuro.
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Position Interpolation for Context Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
YaRN na Context Uburebure
Ibibazo bikunze kubazwa
What is Position Interpolation for Context Extension?
Umwanya Interpolation (PI) ni tekinike irambuye imiterere yururimi rwikoreshwa ryururimi rurenze kure uburebure bwamahugurwa mugukuraho indangagaciro zumwanya aho kuzikuramo. Ireka icyitegererezo cyatojwe kuri, vuga, 2K cyangwa 4K ibimenyetso bifata 32K cyangwa birenga hamwe gusa nu mucyo mwiza.
Niki gitekerezo cyibanze cya Interpolation?
PI igabanya indangagaciro zerekana umwanya wo kwaguka, gushushanya urutonde rurerure rusubira mu gukwirakwiza intera icyitegererezo kimaze kwiga.
Nuwuhe mwanya wa kodegisi ya posisiyo ni Umwanya Interpolation uhujwe cyane?
PI yari ikunzwe cyane kuri moderi ishingiye kuri RoPE, ikuraho impande zizunguruka kugirango zigume mumurongo wamenyerejwe.
Ni ukubera iki naive extrapolation kumurongo muremure ikunda kunanirwa?
Kugaburira ibihe birebire byerekana icyitegererezo kitarenze intera itigeze itoza, bigatuma kwitabwaho no gukora bigabanuka cyane.
Niba kwagura uburebure kuva kuri L kugeza kuri L ', nikihe kintu cyo gukuraho PI shingiro ikoreshwa kumwanya m?
Ikarita ya PI m kugeza m igabanijwe nimpamvu s = L '/ L, igabanya intera ndende mumwanya wambere watojwe.
Nigute NTK-izi gupima na YaRN gutezimbere kuri Interpolation isanzwe?
Ubu buryo buringaniza buke kandi buringaniye muburyo butandukanye, bugumana neza-neza neza umwanya uhagaze neza mugihe ukomeje kugera kumurongo muremure.