YaRN da Tsawon Tsawon yanayi
YaRN (Har yanzu wani karin RoPE) wata ingantacciyar dabara ce don shimfida tagar mahallin abin amfani da samfurin fiye da abin da aka horar da shi.
Dubawa
It cleverly rescales rotary position embeddings so a model trained on, say, 4K tokens can handle 32K or more with minimal fine-tuning.
Zurfafa nutsewa
Yawancin LLMs na zamani suna ɓoye matsayi na alama tare da RoPE (Rotary Position Embeddings), wanda ke juya tambaya da maɓalli na maɓalli ta kusurwoyi masu ɗaure zuwa matsayi. Lokacin da kuke ciyar da jeri fiye da tsayin horo, waɗannan jujjuyawar suna shigar da kewayon da ba a gani kuma ƙirar ta rushe. YaRN, wanda Bowen Peng da masu haɗin gwiwa suka gabatar a cikin 2023, yana gyara wannan tare da NTK-sane interpolation da ake amfani da shi a kowace mita: yana barin maɗaukaki masu girma (waɗanda ke kama alaƙar gida, ɗan gajeren zango) galibi waɗanda ba a taɓa su ba yayin da ke haɗa ƙananan mitoci (wanda ke bin matsayi mai tsayi). YaRN kuma yana ƙara daidaita yanayin zafi zuwa hankali don magance canje-canjen entropy waɗanda ke fitowa daga wurare masu tsayi. Sakamakon yana da ƙarfin aiki na dogon lokaci bayan daidaitawa akan ƙaramin juzu'i na bayanai da matakan da hanyoyin butulci ke buƙata.
Fahimtar Fasaha
RoPE yana sanya kowane girman abin haɗawa mitar juyawa. Matsakaicin madaidaiciyar madaidaiciya yana damfara duk mitoci daidai-da-wane, yana cutar da maɗaukakin mitoci waɗanda ke ɓoye cikakkun bayanai na gida. YaRN yana amfani da aikin ramp ɗin don haɗawa kawai ƙananan mitoci (tsawon tsayin tsayi) yayin da yake adana manyan mitoci, da 1/sqrt (t) yanayin zafin hankali wanda ke kiyaye kaifi mai laushi mai laushi kamar yadda tsayin jeri ke girma. Wannan tsarin NTK-da-ɓangarorin yana faɗaɗa mahallin tare da ƙarancin lalacewa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar YaRN da Tsawon Tsawon yanayi
Tsawaita ma'anar yanzu shine daidaitaccen aiki: buɗaɗɗen samfura suna jigilar bambance-bambancen YaRN akai-akai suna kaiwa alamun 128K ko sama da haka. Bincike yana motsawa zuwa hanyoyin da ke fadada mahallin tare da sifili ko kusa-sifiti mai kyau-daidaita, hada RoPE rescaling tare da dabaru-tsari, da kuma kula da inganci a cikin cikakken taga maimakon kawai ƙarewa. Yi tsammanin haɗakar waɗannan fasahohin cikin horo mai tsawo don haka mahallin ya zama na asali maimakon a sake gyarawa.
Aiwatar da Gaskiyar Duniya
Ƙaddamar da buɗaɗɗen ƙirar yanayi na 4K zuwa 32K ko 128K don doguwar tambaya ta amsa tare da taƙaitaccen daidaitawa.
Ba da damar sake dawo da tsarin haɓaka don shigar da wurare da yawa da aka haɗa ba tare da yanke ba.
Ƙaddamar da mataimakan lamba waɗanda ke buƙatar gabaɗayan babban fayil ɗin ma'ajiya ko fayiloli da yawa a cikin hanzari ɗaya
Daidaita ƙirar tushe don doguwar tattaunawa mai juyawa da yawa waɗanda ke tara manyan tarihin taɗi
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the YaRN and Context Length Extension quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Matsayin Interpolation don Tsawaita Magana
Tambayoyin da ake yawan yi
What is YaRN and Context Length Extension?
YaRN (Har yanzu wani karin RoPE) wata ingantacciyar dabara ce don shimfida tagar mahallin abin amfani da samfurin fiye da abin da aka horar da shi. Yana da wayo yana sake daidaita abubuwan sakawa na jujjuya don haka ƙirar da aka horar da ita, a ce, alamun 4K na iya ɗaukar 32K ko fiye tare da ƙaramin daidaitawa.
Wace hanya ce ta shigar da matsayi YaRN ke gyara don tsawaita mahallin?
YaRN, wanda sunansa ke nufin Duk da haka wani karin RoPE, yana sake daidaita abubuwan sakawa na juyi da aka yi amfani da su a yawancin LLM na zamani.
Me ke faruwa ba daidai ba lokacin da samfurin RoPE ya ga jeri fiye da tsayin horonsa?
Matsayin da ya wuce horo yana haifar da kusurwoyi na juyawa wanda samfurin bai taɓa gani ba, don haka halin kulawa yana raguwa sosai.
Ta yaya YaRN ke kula da nau'ikan mitar RoPE daban-daban?
YaRN yana amfani da ramp na NTK-by-parts wanda ke shiga tsaka-tsaki mai tsayi mai tsayi mai tsayi kuma yana barin gajeriyar matsakaicin matsakaicin matsakaici galibi cikakke.
Bayan sake daidaita mitoci, wane ƙarin gyara ne YaRN ke amfani?
YaRN yana ƙara ƙimar zafin jiki zuwa abubuwan kulawa don magance sauye-sauyen entropy waɗanda ke faruwa yayin da mahallin ke girma.
Menene mabuɗin fa'ida mai amfani na YaRN akan haɗin gwiwar butulci?
YaRN yana samun ingantaccen ingancin yanayi mai tsayi bayan daidaitawa akan ƙaramin juzu'i na hanyoyin butulcin bayanai da ake buƙata.