Ntụle windo Onodu YaRN
YaRN (Ma ọzọ RoPE extension) bụ usoro na-agbatị windo ọnọdụ ihe ntụgharị ihe karịrị nke a zụrụ ya, na-enwe obere nlegharị anya.
Nchịkọta
It matters because it lets existing models handle much longer documents without retraining from scratch.
Ime miri emi
Ọtụtụ LLM nke ọgbara ọhụrụ na-edobe ọnọdụ okwu site na iji Rotary Position Embeddings (RoPE), nke na-arụ ọrụ nke ọma naanị ruo ogologo ihe nlereanya ahụ hụrụ n'oge ọzụzụ. Na-eri nri n'usoro ogologo na ihe nlereanya ahụ na-eweda nke ọma. YaRN na-edozi nke a site n'ịweghachi ugboro ugboro ntụgharị RoPE n'ụzọ mara mma: akụkụ dị elu (nke na-ejide mpaghara, mmekọrịta dị nso) na-emekarị ka ọ ghara imetụ ya aka, ebe akụkụ dị ala (nke na-ejide ọnọdụ ogologo oge) na-ejikọta ya. Ọ na-agbakwụnye mgbanwe mgbanwe okpomọkụ na nlebara anya iji mee ka logits na-akpa àgwà ọma n'ogologo. Nsonaazụ, nke egosipụtara na ụdị LLAMA, gbatịrị okirikiri site na 4K ruo 64K-128K tokens na-eji naanị ihe dịka 0.1% nke data ọzụzụ izizi yana narị ole na ole nlegharị anya usoro.
Nghọta nka nka
RoPE na-atụgharị ajụjụ na vectors igodo site n'akụkụ nhata na ọnọdụ yana ugboro otu akụkụ. Interpolation naive linear (Position Interpolation) na-akụda ugboro niile n'otu aka ahụ, na-emebi nkọwa mpaghara. YaRN na-etinye aka na 'NTK-by-parts': ọ na-ejikọta naanị akụkụ dị ala (ogologo ogologo ogologo), na-ahapụ naanị ndị na-adị elu ma na-agbago n'etiti ha. Nleba anya okpomọkụ na-akwụ ụgwọ maka mgbanwe entropy, na-echekwa izi ezi na ogologo ogologo.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke imegharị windo ọnọdụ YaRN
Mgbatị mara ụdị nke YaRN abụrụla ihe ndabere maka mbupu ụdị ọnọdụ ogologo oge; Ụdị dị iche iche na ndị nọchiri anya na-apụta ka ụlọ nyocha na-aga n'ihu na windo ndenye. Na-atụ anya mwekota siri ike na nlebara anya nke ọma, mkpakọ KV-cache, yana ntule siri ike nke na-edozi n'ụzọ ọ bụla arịrịọ. Ihe na-ewu ewu nke sara mbara bụ n'ịkọwa 'ogologo oge a zụrụ ihe nlereanya' site na 'ogologo oge ọ nwere ike ịgụ nke ọma,' na-eme ka ogologo okwu dị ogologo ka ọ bụrụ atụmatụ ọzụzụ ọzụzụ dị ọnụ ala karịa nkwekọ ụkpụrụ ụlọ dị oke ọnụ.
Mmejuputa n'ezie n'ụwa
Na-agbatị ihe ngosi LLAMA mepere emepe site na 4K ruo akara ngosi 128K ka ọ nwee ike itinye koodubase ma ọ bụ nkwekọrịta ogologo n'otu ngafe.
Ikwe ka onye nkata nkata na-ejigide akụkọ mkparịta ụka ogologo oge na-ewepụghị ntụgharị mbụ
Na-achịkọta akwụkwọ ndị nwere ogologo akwụkwọ ma ọ bụ ndegharị ọtụtụ awa gafere mpio ala nke ihe nlereanya ahụ.
Na-adị ọnụ ala na-emegharị ihe atụ a zụrụ azụ maka ọrụ iweghachite ọnọdụ ogologo oge na-eji naanị obere nlegharị anya.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the YaRN Context Window Scaling quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Windows okwu
Ajụjụ a na-ajụkarị
What is YaRN Context Window Scaling?
YaRN (Ma ọzọ RoPE extension) bụ usoro na-agbatị windo ọnọdụ ihe ntụgharị ihe karịrị nke a zụrụ ya, na-enwe obere nlegharị anya. Ọ dị mkpa n'ihi na ọ na-ahapụ ụdị ndị dị ugbu a ijikwa dọkụmentị dị ogologo karịa na-azụghachighị azụ.
Kedu atụmatụ ntinye ọnọdụ YaRN na-agbanwe iji gbasaa ogologo okwu?
YaRN na-anọchi anya 'Enwekwara ihe ndọtị RoPE ọzọ' ma na-arụ ọrụ site na ịmegharịgharị ugboro ole a na-eji na Rotary Position Embeddings.
Kedu ka YaRN si emeso ugboro dị elu yana akụkụ RoPE dị ala?
Ụzọ 'NTK-site-akụkụ' nke YaRN na-echekwa akụkụ dị elu (mpaghara) ebe ọ na-ejikọta ndị dị obere (ogologo) iji zere imebi nkọwa mpaghara.
Kedu ihe bụ isi uru arụmọrụ YaRN karịa ịzụ ihe nlereanya ogologo oge site na ọkọ?
YaRN nwere ike ịgbatị ọnọdụ ya site na iji ihe dị ka 0.1% nke data ọzụzụ izizi yana ọnụ ọgụgụ dị nta nke nhazi nhazi nke ọma, dị ọnụ ala karịa ịzụghachi azụ.
Ewezuga iwegharị ugboro ugboro, kedu mgbanwe ọzọ YaRN na-etinye iji mee ka nlebara anya ogologo oge kwụsie ike?
YaRN na-agbanwe ọnọdụ nlebara anya iji kwụọ ụgwọ maka mgbanwe entropy/Logit na-eme mgbe usoro na-adịte aka.
Kedu nsogbu na-eme ma ọ bụrụ na ị na-azụ usoro ihe nlereanya RoPE ogologo oge karịa ka a zụrụ ya, na-enweghị ihe ọ bụla?
RoPE n'ozuzu na-adịghị mma na ogologo ọnọdụ a na-ahụghị anya, ya mere ịdịmma na-ada ada ma ọ gwụla ma ewepụghị frequencies.