Uburyo bwo Kwitondera
Icyitonderwa cyemerera icyitegererezo guhitamo andi magambo mumagambo yingirakamaro cyane mugihe asobanura buri jambo.
Incamake
It is the core idea that made the transformer — and therefore modern AI like ChatGPT — possible.
Kwibira cyane
Icyitonderwa gisubiza ikibazo cyoroshye kuri buri jambo: ni ayahe magambo yandi nareba kugirango nsobanukirwe n'iri? Impapuro za 2017 'Icyitonderwa Nicyo Ukeneye cyose' cyanditswe na Vaswani na bagenzi be kuri Google berekanye transformateur, ikoresha ubwitonzi nka moteri nyamukuru kandi igahindura ibishushanyo bishaje. Buri kimenyetso gihindurwamo ibice bitatu: ikibazo (ndashaka iki?), Urufunguzo (ndatanga iki?), N'agaciro (amakuru ntwaye). Ikibazo cyikigereranyo cyagereranijwe nurundi rufunguzo rwibimenyetso kugirango rutange uburemere, hanyuma ruhuza indangagaciro hamwe. Kwiyitaho ubikora muburyo bumwe kugirango buri jambo rishobore kwitabira irindi jambo ryose. Imitwe myinshi-imitwe ikora igereranya ryinshi murwego rumwe, buriwese yibanda kuburyo butandukanye.
Ubushishozi
Imibare yapimwe akadomo-ibicuruzwa byitondewe: softmax (QK ^ T / √d_k) V. Igicuruzwa cyadomo cyibibazo nurufunguzo cyerekana uko buri jambo rifite akamaro; kugabana na kare kare yumuzi wingenzi (√d_k) ituma ayo manota adakura cyane; softmax ibahindura muburemere igereranya imwe; no kugwiza na V bitanga uburemere buremereye bwindangagaciro. Kuberako buri kimenyetso kigereranya nibindi, ikiguzi gikura hamwe na kare yuburebure bwikurikiranya - O (n²) - niyo mpamvu inyongeramusaruro ndende ihenze nimpamvu optimizasiyo nka FlashAttention ibaho.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Ejo hazaza h'uburyo bwo kwitondera
Icyitonderwa kirahari, ariko igiciro cyacyo cya kane gitera ubushakashatsi bukomeye. FlashAtention yatumye ibitekerezo bisanzwe byihuta cyane kandi birushijeho kwibuka-byongeye kubara. Icyerekezo gishya kirimo kwitonderwa gake kandi kumurongo, guhuriza hamwe no kubaza ibibazo byinshi kugirango ugabanye kwibuka mugihe cyibisekuru, hamwe nibishushanyo mbonera bivanga ibitekerezo hamwe na moderi ya leta-umwanya nka Mamba kubintu byinjira cyane. Tegereza sisitemu yigihe kizaza kugirango witondere guhinduka mugihe uhetamye ikiguzi kugirango gutunganya ibitabo-uburebure cyangwa inyandiko nyinshi byinjira biba bisanzwe kandi bihendutse.
Gushyira mu bikorwa Isi
Ubusobanuro bwimashini, aho icyitegererezo cyitabira amagambo yinkomoko mugihe utanga buri jambo ryahinduwe.
Incamake, aho kwitondera bifasha icyitegererezo kwibanda ku nteruro zingenzi mu ngingo ndende.
Abafasha ba kode bitabira ibisobanuro bihinduka mbere yo guhanura umurongo ukurikira.
Ikibazo gisubiza hejuru yinyandiko, aho kwitondera guhuza amagambo yibibazo igice kirimo igisubizo.
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Attention Mechanisms quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Kunyerera Idirishya
Ibibazo bikunze kubazwa
What is Attention Mechanisms?
Icyitonderwa cyemerera icyitegererezo guhitamo andi magambo mumagambo yingirakamaro cyane mugihe asobanura buri jambo. Nigitekerezo cyibanze cyakoze transformateur - nuko rero AI igezweho nka ChatGPT - birashoboka.
Mu nteruro imwe, uburyo bwo kwitondera bukora iki?
Icyitonderwa kibara uburemere bugena umubare buri kimenyetso kigomba gushushanya ku bindi bimenyetso mugihe cyo kugereranya.
Ni izihe mpapuro zerekana 2017 zatangije ubwubatsi bwa transformateur?
'Icyitonderwa nicyo ukeneye byose' (Vaswani et al., 2017) yatanze igitekerezo cyo guhindura ibintu, gishingiye kubitekerezo aho kugirango bisubire.
Nibihe bice bitatu byabazwe kuri buri kimenyetso mubitekerezo?
Buri kimenyetso gitanga ikibazo, urufunguzo, nagaciro; ibibazo bihujwe nurufunguzo rwo kuremera indangagaciro.
Muri formula softmax (QK ^ T / √d_k) V, kuki kugabana na √d_k?
Gupima kwaduka kwadarato yurufunguzo rwibanze birinda ibicuruzwa binini byadusunika softmax mubice bito, bikomeza imyitozo ihamye.
Ni ukubera iki kwiyitaho bisanzwe bihenze kubintu byinjira cyane?
Ikimenyetso cyose cyitabira ikindi kimenyetso, bityo umubare wigereranya umunzani nka n², bigatuma urutonde rurerure ruhenze mugihe no kwibuka.