YaRN na Mgbatị Ogologo Ọdịnaya
YaRN (Ma mgbakwunye RoPE ọzọ) bụ usoro na-arụ ọrụ nke ọma maka ịgbatị mpio ihe ngosi nke ihe nlereanya karịrị nke a zụrụ ya.
Nchịkọta
It cleverly rescales rotary position embeddings so a model trained on, say, 4K tokens can handle 32K or more with minimal fine-tuning.
Ime miri emi
Ọtụtụ LLM nke ọgbara ọhụrụ na-eji RoPE (Rotary Position Embeddings) tinye ọnọdụ akara ngosi, nke na-atụgharị ajụjụ na vectors isi site n'akụkụ kekọtara na ọnọdụ. Mgbe ị na-eri usoro ogologo oge karịa ogologo ọzụzụ, ntụgharị ndị a na-abanye n'ọsọ ndị a na-adịghị ahụ anya na ihe nlereanya ahụ daa. YaRN, ewepụtara na 2023 site na Bowen Peng na ndị na-emekọ ihe ọnụ, na-edozi nke a na NTK-aware interpolation etinyere n'otu oge: ọ na-ahapụ akụkụ dị elu (nke na-ejide mmekọrịta mpaghara, nke dị mkpụmkpụ) nke emetụghị aka mgbe ọ na-ejikọta akụkụ dị ala (nke na-agbaso ọnọdụ ogologo oge). YaRN na-agbakwụnye ngbanwe ọnọdụ okpomọkụ na nlebara anya iji gbochie mgbanwe entropy na-abịa site na ọnọdụ dị ogologo. Nsonaazụ bụ arụmọrụ ogologo oge siri ike ka emezigharị nke ọma na naanị ntakịrị ntakịrị nke data yana usoro ndị ụzọ enweghị isi chọrọ.
Nghọta nka nka
RoPE na-ekenye akụkụ ntinye ọ bụla ugboro ntụgharị. Interpolation naive linear na-akpakọba ugboro niile n'otu aka ahụ, na-emebi akụkụ ugboro ugboro dị elu nke na-etinye nkọwa mpaghara dị mma. YaRN na-eji ọrụ rampụ na-atụgharị naanị akụkụ dị ala (ogologo ogologo ogologo) ka ọ na-echekwa ndị dị elu, gbakwunyere 1/sqrt (t) nlebara anya ọnọdụ okpomọkụ nke na-eme ka ịdị nro softmax kwụsie ike ka ogologo usoro na-eto. Usoro NTK-site-akụkụ a na-agbatị ọnọdụ ya na obere mmebi.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke YaRN na Mgbatị Ogologo Ọdịnihu
Mgbatị ihe dị ugbu a bụ ụkpụrụ ọkọlọtọ: ụdị mepere emepe na-ebufe YaRN variants agbatị na-erute akara ngosi 128K ma ọ bụ gafere. Nchọpụta na-aga n'ihu na usoro na-agbatị gburugburu ya na nlegharị anya efu ma ọ bụ nso-efu, jikọta RoPE rescaling na usoro nlebara anya, ma nọgide na-adị mma n'ofe windo zuru ezu karịa naanị njedebe. Na-atụ anya mwekota nke usoro ndị a n'ịkwalite ọzụzụ ogologo oge nke mere na gburugburu ebe obibi bụ nke ala kama ịmegharịgharị ya.
Mmejuputa n'ezie n'ụwa
Na-agbatị ihe ngosi 4K-emepe emepe gaa na 32K ma ọ bụ 128K maka ajụjụ ogologo akwụkwọ na-aza na nhazi nkenke.
Na-eme ka sistemu agbakwunyere iweghachite ka ị webata ọtụtụ akụkụ akọrọ ọnụ na-enweghị mkpọpu.
Ndị enyemaka koodu na-achọ nnukwu faịlụ nchekwa ma ọ bụ ọtụtụ faịlụ n'otu ngwa ngwa
Ịmegharị ụkpụrụ ntọala maka mkparịta ụka ọtụtụ ntụgharị ogologo na-achịkọta nnukwu akụkọ nkata
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the YaRN and Context Length Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nkwekọrịta Ọnọdụ maka Mgbatị Ọdịnihu
Ajụjụ a na-ajụkarị
What is YaRN and Context Length Extension?
YaRN (Ma mgbakwunye RoPE ọzọ) bụ usoro na-arụ ọrụ nke ọma maka ịgbatị mpio ihe ngosi nke ihe nlereanya karịrị nke a zụrụ ya. Ọ na-eji akọ weghachi ihe ntinye ọnọdụ rotary ka ihe atụ a zụrụ azụ na, sịnụ, akara ngosi 4K nwere ike ijikwa 32K ma ọ bụ karịa site na iji obere nlegharị anya.
Kedu usoro ntinye koodu nke YaRN na-agbanwe iji gbatịa okirikiri?
YaRN, onye aha ya pụtara Mgbakwunye RoPE ọzọ, na-eweghachite ntinye ọnọdụ rotary ejiri na ọtụtụ LLM ọgbara ọhụrụ.
Kedu ihe na-ezighi ezi mgbe ụdị RoPE na-ahụ usoro ogologo karịa ogologo ọzụzụ ya?
Ọnọdụ gafere ọzụzụ na-arụpụta akụkụ ntụgharị nke ihe nlereanya ahụ ahụtụbeghị, yabụ omume nlebara anya na-eweda nke ọma.
Kedu otu YaRN si emeso akụkụ ugboro ole RoPE dị iche iche?
YaRN na-eji NTK-site-akụkụ ramp nke na-ejikọta ogologo-wavelength nke dị ala na-eme ka ọ ghara ịdị mkpụmkpụ ma na-ahapụ mkpirisi mkpirisi dị elu na-adịkarịghị.
E wezụga iweghachite ugboro ole, kedu mmezi ọzọ YaRN na-etinye?
YaRN na-agbakwụnye mkpali okpomọkụ na nlebara anya nlebara anya iji gbochie mgbanwe entropy na-eme ka ọnọdụ na-eto eto.
Kedu ihe bụ isi uru bara uru nke YaRN karịa interpolation naive?
YaRN na-enweta ogo ọnọdụ ogologo oge siri ike ka emezigharịrị nke ọma na obere akụkụ nke ụzọ enweghị isi data chọrọ.