Ọnọdụ ALiBi
ALiBi (nlebara anya na Linear Biases) bụ ụzọ dị nkọ iji nye ndị na-agbanwe agbanwe echiche nke usoro okwu na-enweghị ntinye ọnọdụ ọdịnala.
Nchịkọta
It lets a model trained on short text handle much longer inputs at inference time.
Ime miri emi
Ndị na-agbanwe agbanwe enweghị echiche etinyere n'usoro usoro okwu, yabụ ha chọrọ ụzọ ha ga-esi tinye koodu n'ọkwa. Usoro kpochapụwo na-agbakwunye ntinye ọnọdụ na vector token. ALiBi, nke Press, Smith na Lewis webatara na 2021, tụpụrụ ndị ahụ kpamkpam. Kama, ọ na-ewetu akara nlebara anya ozugbo: mgbe ihe nrịbama ajụjụ lere anya na akara igodo, ALiBi na-ewepụ ntaramahụhụ dịka anya n'etiti ha. Ihe nrịbama ndị dịpụrụ adịpụ na-enweta ntaramahụhụ ka ukwuu, yabụ ihe nlereanya ahụ na-amasịkarị ndị gbara ya gburugburu. Isi nlebara anya nke ọ bụla na-enweta mkpọda ntaramahụhụ nke ya, yabụ isi ụfọdụ na-ele anya na mpaghara ebe ndị ọzọ na-ahụ anya. N'ihi na ele mmadụ anya n'ihu bụ naanị ọrụ dị anya, ALiBi na-eji amara mee ka usoro dị ogologo karịa nke a hụrụ na ọzụzụ.
Nghọta nka nka
Maka ajụjụ n'ọnọdụ i na igodo na ọnọdụ j, ALiBi na-agbakwụnye m * (j - i) na akara nlebara anya raw tupu softmax, ebe m bụ isi-kpọmkwem mgbe niile (mkpọda ugwu na-etolite usoro geometric dị ka 1/2, 1/4, 1/8). Ebe j bụ ihe na-erughị ma ọ bụ ha nhata na nlebara anya ihe kpatara ya, okwu a bụ efu ma ọ bụ adịghị mma, na-ata akara ngosi dị anya. Ọnweghị paramita mmụta na enweghị mgbakwunye agbakwunyere, yabụ naanị ihe dị n'elu bụ matriks enweghị isi agbagoro agbagoro.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke ALiBi Ọnọdụ Bias
ALiBi gosipụtara na ndị ikwu, nhụsianya dabere n'ebe dị anya na-akụri ihe ntinye ọnọdụ zuru oke maka ịkọwapụta ogologo oge, na echiche ahụ na-emetụta atụmatụ ogologo oge ọgbara ọhụrụ. Offọdụ ụdị na-adịbeghị anya na-akwado mgbakwunye rotary (RoPE) kama, mana ALiBi ka na-ewu ewu ebe oke mgbakwunye dị mkpa ma jiri ya mee ihe n'ụdị dịka BLOOM na MPT. Na-atụ anya nnwale ngwakọ na-aga n'ihu, na-ejikọta echiche dị anya na nchacha RoPE, ka ụlọ nyocha na-akwali windo ndị gbara gburugburu n'akụkụ ọtụtụ nde token na-azụghachighị azụ.
Mmejuputa n'ezie n'ụwa
Ọzụzụ chatbot na ihe atụ 1,024-token mana ibuga ya na akwụkwọ 4,096-token na-enweghị ọzụzụ ọzọ, dabere na mgbakwunye ALiBi.
Ụdị asụsụ ọtụtụ BLOOM 176B, nke nakweere ALiBi maka njikwa ọnọdụ ya.
MosaicML's MPT ụdị, nke ji ALiBi kpọsaa nke ọma ogologo ọnọdụ gburugburu na-akparaghị ókè na ntụle.
Na-achịkọta ogologo nkwekọrịta iwu nke gafere ogologo ọzụzụ izizi nke ihe nlereanya ahụ, ebe nhụsianya dị nso na-eme ka nlebara anya na-adakọ ọnụ.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the ALiBi Position Bias quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nkwekọrịta Ọnọdụ maka Mgbatị Ọdịnihu
Ajụjụ a na-ajụkarị
What is ALiBi Position Bias?
ALiBi (nlebara anya na Linear Biases) bụ ụzọ dị nkọ iji nye ndị na-agbanwe agbanwe echiche nke usoro okwu na-enweghị ntinye ọnọdụ ọdịnala. Ọ na-ahapụ ihe nlereanya a zụrụ n'edemede dị mkpụmkpụ ijikwa ntinye ogologo ogologo oge n'oge ntinye.
Kedu ihe ALiBi pụtara?
ALiBi dị mkpụmkpụ maka Ntị na Linear Biases, aha ya maka ntaramahụhụ ahịrị ọ na-agbakwunye na akara nlebara anya.
Kedu ka ALiBi si ata ahụhụ ndị dị anya?
ALiBi wepụrụ uru dabara na anya igodo ajụjụ site na akara nlebara anya, yabụ akara ngosi ga-adịwanye ibu.
Kedu uru bara uru ALiBi kacha eme mmemme?
N'ihi na nhụsianya ahụ na-adabere naanị n'ebe dị anya, ụdị a zụrụ n'usoro dị mkpirikpi nwere ike ijikwa ndị dị ogologo karịa n'oge ntinye.
Kedu ihe dị iche na nhụsianya ahịrị n'ofe nlebara anya isi?
ALiBi na-ekenye isi nke ọ bụla otu mkpọda dị iche iche (dịka, 1/2, 1/4, 1/8), ya mere ndị isi dị iche iche na-aga na ọkwa dị iche iche.
E jiri ya tụnyere ntinye ọnọdụ ọdịnala, ALiBi na-agbakwunye nkeji ole amụtara?
ALiBi ewebataghị usoro mmụta ọhụrụ; mkpọda nke kwa-isi bụ ndị a kara aka na-adịgide adịgide.