ALiBi Umwanya Kubogama
ALiBi.
Incamake
It lets a model trained on short text handle much longer inputs at inference time.
Kwibira cyane
Abahindura ntabwo bafite ibitekerezo byubaka byurutonde, bityo bakeneye inzira yo gushiraho umwanya. Uburyo bwa kera bwongeramo imyanya kuri token vectors. ALiBi, yatangijwe na Press, Smith, na Lewis mu 2021, yirukana abo rwose. Ahubwo, yerekana amanota yibitekerezo bitaziguye: mugihe ikibazo cyibibazo kireba urufunguzo, ALiBi ikuramo igihano kijyanye nintera iri hagati yabo. Tokens iri kure cyane ibona igihano kinini, kuburyo moderi isanzwe ikunda imiterere iri hafi. Buri mutwe wokwitondera ubona igihano cyacyo cyagenwe, kuburyo imitwe imwe ireba mugace mugihe abandi babona kure. Kuberako kubogama nigikorwa cyintera gusa, ALiBi isimburana neza muburyo bukurikiranye kurenza uko bigaragara mumahugurwa.
Ubushishozi
Kubibazo kumwanya i nurufunguzo kumwanya wa j, ALiBi yongeraho m * (j - i) kumanota yibanze mbere ya softmax, aho m ari umutwe wihariye uhoraho (ahahanamye hagizwe na geometrike ikurikirana nka 1/2, 1/4, 1/8). Kubera ko j ari munsi cyangwa ingana na i mubitekerezo byimpamvu, iri jambo ni zeru cyangwa ribi, rihana ibimenyetso bya kure. Nta bipimo byize kandi nta gushiramo byongeweho, bityo hejuru yonyine ni matrike ibogamye.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza ka ALiBi Umwanya Kubogama
ALiBi yerekanye ko isano, intera ishingiye kubogamye ikubita umwanya wuzuye gushiramo uburebure rusange, kandi icyo gitekerezo ubu cyinjiye mubishushanyo mbonera bigezweho. Moderi zimwe ziheruka zishira muburyo bwo kuzenguruka (RoPE) aho, ariko ALiBi ikomeza gukundwa cyane aho ikibazo cya extrapolation gikabije kandi cyakoreshejwe mubyitegererezo nka BLOOM na MPT. Witege ko hakomeza kugeragezwa kuvangwa, guhuza kubogama kwintera hamwe no gupima RoPE, mugihe laboratoire isunika Windows yerekeza kuri miriyoni yikimenyetso utarinze kwitoza.
Gushyira mu bikorwa Isi
Guhugura chatbot ku ngero 1.024-token ariko kuyishyira ku nyandiko 4.096-yerekana ibimenyetso utabanje kwitoza, wishingikirije kuri ALiBi yoherezwa hanze.
BLOOM 176B yerekana indimi nyinshi, yemeye ALiBi kugirango ikore imyanya yayo.
Moderi ya MPT ya MosaicML, yakoresheje ALiBi kugirango yamamaze neza uburebure butagira imipaka.
Incamake yamasezerano maremare yemewe arenze icyitegererezo cyambere cyamahugurwa, aho hafi-imiterere ibogamye ikomeza kwitondera.
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the ALiBi Position Bias quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Umwanya Interpolation yo Kwagura Imirongo
Ibibazo bikunze kubazwa
What is ALiBi Position Bias?
ALiBi. Kureka icyitegererezo cyatojwe kumyandiko migufi ikora inyongeramusaruro ndende mugihe cyo gufata umwanzuro.
ALiBi igereranya iki?
ALiBi ni ngufi yo Kwitonda hamwe na Linear Biases, yitiriwe ibihano byumurongo byongera kumanota yo kwitondera.
Nigute ALiBi ihana ibimenyetso bya kure?
ALiBi ikuramo agaciro gahwanye nikibazo-urufunguzo ruri hagati y amanota yo kwitabwaho, bityo ibimenyetso bya kure bikagira uburemere buke.
Ni izihe nyungu za ALiBi zizwi cyane?
Kuberako kubogama biterwa nintera gusa, moderi yatojwe kumurongo mugufi irashobora gukora igihe kirekire mugihe cyo gufata umwanzuro.
Ni irihe tandukaniro riri hagati yo kubogama kumutwe?
ALiBi igenera buri mutwe ahantu hahanamye, hahanamye (urugero, 1/2, 1/4, 1/8), bityo imitwe itandukanye yitabira kurwego rutandukanye.
Ugereranije no gushyiramo imyanya gakondo, ALiBi yongeraho ibipimo bingahe byize?
ALiBi itangiza ibipimo bishya byize; ahahanamye kuri buri mutwe hateganijwe mbere.