Nkwekọrịta ọnọdụ maka ogologo okwu
Interpolation Ọnọdụ (PI) bụ usoro dị mfe, nke nwere mmetụta nke na-agbatị windo ọnọdụ nke Transformer site na ịpịnye indices ọnọdụ ọhụrụ n'ime oke ihe nlereanya ahụ maralarị.
Nchịkọta
Instead of extrapolating to unseen positions, it interpolates within trained ones, requiring only brief fine-tuning.
Ime miri emi
Ndị nyocha Meta webatara (Chen et al.) na 2023, Interpolation Positional na-akwado eziokwu ahụ bụ na ụdị ndị nwere RoPE na-ada n'ike n'ike mgbe ha na-ebupụ n'ọkwa gafere ọzụzụ. Nghọta a na-emegiderịta onwe ya: kama ịrịọ ihe nlereanya ahụ ka ọ na-ejikwa ụkpụrụ ọnọdụ dị ukwuu ọ na-ahụtụbeghị, PI na-ekewa indices ọnọdụ na-abata site na ihe dị elu ka ogologo nke, sịnụ, 8K maapụ laghachi azụ na mbụ 2K nso. N'ihi na a zụrụ ihe nlereanya ahụ na nso nso a, ntụgharị ndị ahụ na-anọ na nkesa. Mgbe naanị 1,000 nhazi nhazi nke ọma, ihe nlereanya LLAMA gbatịpụrụ n'ụzọ a ka ejiri mee ihe ruo 32K. Akwụkwọ akụkọ ahụ gosiri extrapolation nwere ike ime ka nlebara anya n'ọtụtụ buru ibu, ebe interpolation na-eme ka ha nwee oke ma kwụsie ike, ya mere interpolation na-arụ ọrụ nke ọma karịa mgbakwunye.
Nghọta nka nka
PI rescale m ka m/s ebe s bụ ihe ndọtị (dịka ọmụmaatụ, ogologo ọhụrụ kewara site na ogologo mbụ). Maka RoPE nke a na-ebelata nke ọma ntụgharị ntụgharị n'etiti ọnọdụ ndị dị n'akụkụ, na-akwakọba ọnọdụ karịa n'ime akụkụ akụkụ a zụrụ azụ. The theoretical bounded in the papers gosiri interpolated nlebara anya akara na-anọ nke ọma na-achịkwa, ebe naive extrapolation nwere ike na-emepụta scores iwu nke ịdị ukwuu karịa ihe ọ bụla hụrụ na ọzụzụ, destabilizing softmax.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke njikọta ọnọdụ maka ogologo okwu
Interpolation ọnọdụ ghọrọ ntọala maka ebili mmiri nke nsonye, gụnyere NTK-aware scaling na YaRN, nke na-emekọ ọnụ karịa iji chekwaa nkọwa mpaghara. Usoro a bụ maka usoro ndị chọrọ ntakịrị ma ọ bụ na-enweghị nhazigharị nke ọma yana n'ịme ihe njikwa ogologo oge n'ime ọzụzụ ọzụzụ. PI ka bụ ntọala bara uru ma na-ejikọta ya na atụmatụ mara ugboro ugboro iji ruo 128K-gbakwunyere okirikiri windo nke ọma.
Mmejuputa n'ezie n'ụwa
Na-agbatị ụdị LLMA nke 2K-context iji jikwaa akara ngosi 8K-32K nwere ihe dị ka 1,000 ezigbo nhazi usoro.
Ịmegharị ụdị nkata dị adị maka nchịkọta akwụkwọ ogologo oge na-azụghachighị azụ
Na-eje ozi dị ka ntọala echiche nke NTK-maara scaling na YaRN na-emeziwanye
Na-enyere koodu ederede ogologo oge ma ọ bụ nyocha akwụkwọ iwu na ụdị ejiri windo mkpụmkpụ zụrụ nke mbụ
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Positional Interpolation for Long Context quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nkwekọrịta Ọnọdụ maka Mgbatị Ọdịnihu
Ajụjụ a na-ajụkarị
What is Positional Interpolation for Long Context?
Interpolation Ọnọdụ (PI) bụ usoro dị mfe, nke nwere mmetụta nke na-agbatị windo ọnọdụ nke Transformer site na ịpịnye indices ọnọdụ ọhụrụ n'ime oke ihe nlereanya ahụ maralarị. Kama ịpụpụ n'ọnọdụ ndị a na-adịghị ahụ anya, ọ na-emekọrịta n'etiti ndị a zụrụ azụ, na-achọ naanị nhazi nkenke.
Kedu ihe bụ isi echiche dị n'azụ Interpolation Positional?
PI na-ekewa indices ọnọdụ site n'ọ̀tụ̀tụ̀ ọ̀tụ̀tụ̀ dị ogologo nke mere ka ogologo usoro dị ogologo maapụ azụ n'ime ebe a zụrụ azụ, na-edobe ntụgharị na nkesa.
Kedu ihe kpatara naive extrapolation na ogologo ọnọdụ ji ada?
Akwụkwọ PI gosipụtara nnukwu ọnọdụ ndị a na-adịghị ahụ anya nwere ike iwepụta nlebara anya nlebara anya n'ọtụtụ dị ukwuu karịa ọzụzụ, na-emebi softmax.
Kedu ihe nlegharị anya nke ọma ka ọrụ PI mbụ chọrọ ịgbatị LLAMA ruo 32K?
Akwụkwọ akụkọ ahụ kọrọ na ihe dị ka 1,000 usoro nlegharị anya nke ọma zuru ezu iji gbatịa gburugburu ruo akara ngosi 32K.
Ọ bụrụ na ịgbatị ọnọdụ okwu site na ihe s, kedu ka PI si agbanwe ọnọdụ m?
PI na-eweghachi ọnọdụ m ruo m/s, na-akpakọba oke ọhụrụ ka ukwuu n'ime usoro ọzụzụ izizi.
Kedu ka PI siri metụta usoro ndị ọzọ dịka YaRN?
PI guzobere interpolation dị ka ụzọ nchekwa; NTK maara ihe na YaRN nụchara anụcha ya site n'ịtụgharị ugboro ugboro karịa.