Nkwekọrịta Ọnọdụ maka Mgbatị Ọdịnihu
Interpolation ọnọdụ (PI) bụ usoro na-agbatị windo ọnọdụ ihe atụ nke asụsụ a na-eji eme ihe karịrị ogologo ọzụzụ ya site n'ịmegharị indices ọnọdụ kama ịwepụ ha.
Nchịkọta
It lets a model trained on, say, 2K or 4K tokens handle 32K or more with only light fine-tuning.
Ime miri emi
Ọtụtụ LLM nke ọgbara ọhụrụ na-eji ntinye ọnọdụ rotary (RoPE), nke na-edekọ ọnọdụ dịka akụkụ ntụgharị etinyere na ajụjụ na isi vectors. Ọ bụrụ na ị na-eri nri ogologo usoro, ihe nlereanya ahụ na-ahụ ọnọdụ na akụkụ ntụgharị ọ na-azụbeghị na ya, na arụmọrụ na-ada n'ihi na nlebara anya na-adịghị mma na ugboro ole na-adịghị. Interpolation ọnọdụ na-ezere extrapolation: ịgbatị site n'ogologo L ruo ogologo L', ọ na-ekewa nrịbama ọnọdụ ọ bụla site na ihe L'/L, na-amanye ihe ọhụrụ ahụ azụ n'ime oge a zụrụ azụ. Ihe nlereanya ugbu a na-ahụ naanị akụkụ nkesa, naanị agbasawanye karịa. Egwu dị mkpụmkpụ (nke na-abụkarị nzọụkwụ narị ole na ole ruo otu puku) na-ahapụ ya ka ọ kwekọọ na oghere kacha mma, na-emepụta omume ogologo oge kwụsiri ike na ntakịrị ntakịrị nke ọnụ ahịa ọzụzụ ọzụzụ.
Nghọta nka nka
RoPE na-atụgharị akụkụ abụọ n'ogo ugboro ole na ole na-adị mma ruo n'ike. PI na-eweghachi ọnọdụ m ruo m/s ebe s = L'/L, ya mere akụkụ ntụgharị na-anọ n'ime ebe a zụrụ azụ kama ịpụpụ. Ụdị dị iche iche mara ugboro ugboro dị ka NTK-aware scaling na YaRN na-aga n'ihu: ha na-abawanye obere ugboro ole na ole na elu frequencies karịa (ma ọ bụ na-ejikọta ya na ogologo ogologo ogologo), na-echekwa nkọwa mpaghara dị elu ma na-agbatị ogologo oge ogologo oge.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke Interpolation Ọnọdụ maka Mgbatị Ọdịnihu
Mgbatị okwu na-aga ngwa ngwa. Ụzọ dị ka NTK-aware RoPE scaling, YaRN, na dynamic/long-RoPE ugbu a na-agbanye windo ruo ọtụtụ narị puku ma ọ bụ ọbụna ọtụtụ nde token, mgbe ụfọdụ na-enweghị obere nhazi ma ọ bụ enweghị mma. Na-atụ anya ka ejikọta aghụghọ ndị a na nlebara anya nke ọma yana mkpakọ KV-cache, yana ka ọ bụrụ akara ọkọlọtọ na nhazi ụdị. Nchọpụta na-aga n'ihu na idobe izi ezi elu n'ofe windo zuru oke ka ogologo ọnọdụ dị n'ime ya nwere ike iji ya eme ihe, ọ bụghị naanị n'aha akwadoro.
Mmejuputa n'ezie n'ụwa
Na-agbatị ihe nlereanya LLMA zụrụ nke 4K gaa na ọnọdụ 32K iji chịkọta ogologo akwụkwọ ka emechara nkenke ọma.
Na-ebunye koodu ntọala dum ma ọ bụ nnukwu nkwekọrịta iwu n'ime otu ngwa ngwa maka azịza ajụjụ obe.
Iji NTK-aware ma ọ bụ YaRN scaling ka ịgbatị ọnọdụ gburugburu na obere ma ọ bụ enweghị ọzụzụ ọzọ.
Na-eje ozi ogologo mkparịta ụka akụkọ ihe mere eme na-enweghị mkpọpu site n'iweghachite ọnọdụ RoPE n'oge ntinye.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Position Interpolation for Context Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
YaRN na Mgbatị Ogologo Ọdịnaya
Ajụjụ a na-ajụkarị
What is Position Interpolation for Context Extension?
Interpolation ọnọdụ (PI) bụ usoro na-agbatị windo ọnọdụ ihe atụ nke asụsụ a na-eji eme ihe karịrị ogologo ọzụzụ ya site n'ịmegharị indices ọnọdụ kama ịwepụ ha. Ọ na-ahapụ ihe nlereanya a zụrụ azụ, sịnụ, 2K ma ọ bụ 4K token jikwaa 32K ma ọ bụ karịa na-eji naanị nlegharị anya dị mma.
Gịnị bụ isi echiche nke Ọnọdụ Interpolation?
PI na-ekewa indices ọnọdụ site na ihe ndọtị, na-esepụta ogologo usoro azụ azụ n'ime oke nkesa nke ahụ mụtaburu.
Kedu atụmatụ ngbanwe ọnọdụ ọnọdụ ka jikọtara ya na ya?
A na-ewu ewu PI maka ụdị dabere na RoPE, na-emegharị akụkụ ntụgharị ka ha nọrọ n'ime ugboro ole a zụrụ azụ.
Kedu ihe kpatara mwepu enweghị uche gaa na ọnọdụ dị ogologo na-ada ada?
Inye usoro nri ogologo oge na-ekpughe ihe nlereanya ahụ n'akụkụ akụkụ ndị na-abụghị nke a zụrụ azụ na ya, na-eme ka nlebara anya na ịrụ ọrụ weda nke ọma.
Ọ bụrụ na ịgbatị ogologo okirikiri site na L ruo L', kedu ihe mweghachi nke PI bụ isi na-emetụta ọnọdụ m?
Maapụ PI m ruo m kewara site na isi ihe s = L'/L, na-akpakọ ogologo ogologo n'ime oge ọzụzụ izizi.
Kedu ka NTK-maara scaling na YaRN si emeziwanye n'ọkwa dị larịị?
Ụzọ ndị a na-abawanye ugboro dị ala na nke dị elu dị iche iche, na-ejigide nkọwa ọnọdụ mpaghara mara mma ma ka na-eru ogologo ọnọdụ.