Ubuyobozi bwa tekiniki

Umwanya wa Interpolation kumwanya muremure

Umwanya wa Interpolation (PI) nubuhanga bworoshye, bukomeye bwagura idirishya ryimiterere ya Transformer ukanda indangagaciro nshya mumurongo icyitegererezo kimaze kumenya.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Kazoza ka Interpolation ya Position kumwanya muremure
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

Instead of extrapolating to unseen positions, it interpolates within trained ones, requiring only brief fine-tuning.

Kwibira cyane

Yatangijwe na Meta abashakashatsi (Chen et al.) Mu 2023, Positional Interpolation ikemura ikibazo cyuko moderi hamwe na RoPE zananirana bikabije mugihe zoherejwe mumyanya irenze amahugurwa. Ubushishozi buvuguruzanya: aho gusaba icyitegererezo gukora indangagaciro nini itigeze ibona, PI igabanya ibipimo byinjira byinjira ku gipimo cyerekana igipimo cyerekana uburebure bwa, kuvuga, amakarita 8K asubira mu mwimerere wa 2K. Kuberako icyitegererezo cyatojwe kururwo rwego, kuzunguruka biguma-kugabura. Nyuma yintambwe 1.000 gusa yo gutunganya neza, moderi ya LLaMA yaguye ubu buryo ikemura kugeza 32K. Uru rupapuro rwerekanye ko extrapolation ishobora gutesha agaciro amanota manini, mugihe interpolation ituma imipaka kandi itajegajega, niyo mpamvu interpolation ikora neza cyane kuruta kurenganurwa.

Gushyira mu bikorwa Isi

Kwagura moderi ya 2K-ya LLaMA kugirango ikore ibimenyetso 8K-32K hamwe nintambwe zigera ku 1.000

Guhuza uburyo bwo kuganira buriho kubwinyandiko ndende incamake utarinze kwitoza

Gukora nkibisobanuro fatizo NTK-izi gupima na YaRN itezimbere

Gushoboza imirongo miremire cyangwa amategeko-inyandiko yisesengura kuri moderi yabanje gutozwa na Windows ngufi

Kazoza ka Interpolation ya Position kumwanya muremure

Umwanya wa Interpolation wabaye ishingiro ryumurongo wo gukurikirana, harimo NTK-uzi gupima na YaRN, ihuza cyane guhitamo kugirango ibungabunge amakuru yaho. Inzira nyabagendwa yerekeza kuburyo bukenera bike cyangwa budahuza neza no guteka ibintu birebire bikoreshwa mukwitegura. PI ikomeza kuba ibyingenzi kandi ikunze guhuzwa na gahunda nshya-yamenyeshejwe gahunda yo kugera kuri 128K-wongeyeho imiterere ya Windows neza.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Positional Interpolation for Long Context quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is Positional Interpolation for Long Context?

Umwanya wa Interpolation (PI) nubuhanga bworoshye, bukomeye bwagura idirishya ryimiterere ya Transformer ukanda indangagaciro nshya mumurongo icyitegererezo kimaze kumenya. Aho gutondekanya imyanya itagaragara, irahuza hagati yabatojwe, bisaba gusa guhuza neza.

Ni ikihe gitekerezo cyibanze inyuma ya Interpolation?

PI igabanya ibipimo byerekana umwanya munini kuburyo ikarita ndende ikurikirana ikagaruka mumyitozo yatojwe, igakomeza kuzenguruka mukwirakwiza.

Ni ukubera iki naive extrapolation kumwanya muremure byananirana?

Urupapuro rwa PI rwerekanye imyanya minini itagaragara irashobora gutanga amanota yo gutondekanya ubunini burenze amahugurwa, guhungabanya softmax.

Nkanswe kangahe-gutunganya neza akazi ka mbere PI yasabye kwagura LLaMA kugeza 32K?

Uru rupapuro rwatangaje ko intambwe zigera ku 1.000 zo gutunganya neza zari zihagije kugira ngo ibintu bigere kuri 32K.

Niba waguye imiterere kubintu s, nigute PI ihindura umwanya m?

PI ikuraho umwanya m kugeza m / s, ikanda intera nini nini murwego rwambere rwahuguwe.

Nigute PI yagize ingaruka muburyo bwanyuma nka YaRN?

PI yashyizeho interpolation nkuburyo bwizewe; NTK-izi gupima na YaRN yarayinonosoye muguhuza imirongo inshuro nyinshi.