Usoro nlebara anya
Nlebara anya na-eme ka ihe nlere anya kpebie okwu ndị ọzọ dị n'ahịrịokwu kacha karịa mgbe a na-atụgharị okwu ọ bụla.
Nchịkọta
It is the core idea that made the transformer — and therefore modern AI like ChatGPT — possible.
Ime miri emi
Nlebara anya na-aza ajụjụ dị mfe maka okwu ọ bụla: kedu okwu ọzọ m kwesịrị ileba anya iji ghọta nke a? Akwụkwọ 2017 'Ntị bụ ihe niile ị chọrọ' nke Vaswani na ndị ọrụ ibe ya na Google webatara transformer, nke na-eji nlebara anya dị ka injin ya bụ isi wee tụfuo atụmatụ ochie na-emegharị ugboro ugboro. A na-atụgharị akara ọ bụla ka ọ bụrụ vector atọ: ajụjụ (gịnị ka m na-achọ?), igodo (gịnị ka m na-enye?), na uru (ozi m na-ebu). A na-atụnyere ajụjụ token megide igodo akara ndị ọzọ iji mepụta nha nlebara anya, nke na-agwakọta ụkpụrụ ọnụ. Nlebara anya onwe onye na-eme nke a n'ime otu usoro ka okwu ọ bụla wee banye ozugbo na okwu ọ bụla ọzọ. Nleba anya ọtụtụ isi na-eme ọtụtụ ntụnyere dị otu a, nke ọ bụla na-elekwasị anya n'ụdị dị iche iche.
Nghọta nka nka
The mgbakọ na mwepụ bụ scaled ntụpọ-ngwaahịa nlebara anya: softmax(QK^T / √d_k) V. The ntụpọ ngwaahịa nke ajụjụ na igodo akara ka mkpa ọ bụla ụzọ dị; Nkewa site na mgbọrọgwụ square nke akụkụ igodo (√d_k) na-eme ka akara ndị ahụ ghara ito oke ibu; softmax na-atụgharị ha ka ha bụrụ nha nke na-agbakọta na otu; na ịmụba site na V na-emepụta ngwakọta dị arọ nke ụkpụrụ. N'ihi na akara ngosi ọ bụla na-atụnyere onye ọ bụla ọzọ, ọnụ ahịa na-eto na square nke usoro ogologo - O (n²) - nke mere ntinye ogologo ji dị oke ọnụ yana ihe kpatara njikarịcha dị ka FlashAttention dị.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Usoro nlebara anya n'ọdịnihu
Nlebara anya dị ebe a ka ọ nọrọ, mana ọnụ ahịa quadratic ya na-akpali nyocha siri ike. FlashAttention mere nlebara anya ọkọlọtọ ngwa ngwa na ebe nchekwa karịa nke ọma site n'ịhazigharị mgbakọ ahụ. Ntuziaka ọhụrụ na-agụnye nlebara anya na-adịghị ahụkebe na nke ahịrị, nleba anya nke mkpokọta na ọtụtụ ajụjụ iji belata ebe nchekwa n'oge ọgbọ, yana ụdị ngwakọ na-agwakọta uche na ụdị oghere steeti dịka Mamba maka ntinye ogologo oge. Na-atụ anya ka usoro n'ọdịniihu dobe mgbanwe nlebara anya ka ọ na-ehulata usoro ọnụ ahịa ka nhazi ogologo akwụkwọ ma ọ bụ ntinye ọtụtụ akwụkwọ wee bụrụ ihe na-aga n'ihu ma dịkwa ọnụ ala.
Mmejuputa n'ezie n'ụwa
Ntụgharị asụsụ igwe, ebe ihe nlere anya na-aga na okwu isi mmalite dị mkpa mgbe ị na-emepụta mkpụrụokwu ọ bụla atụgharịrị.
Nchịkọta, ebe nlebara anya na-enyere ihe nlereanya ahụ aka ilekwasị anya n'ahịrịokwu kachasị mkpa n'ime isiokwu dị ogologo.
Ndị na-enyere aka koodu na-aga azụ na nkọwa agbanwe agbanwe na mbụ mgbe ha na-ebu amụma ahịrị na-esote.
Azịza ajụjụ n'elu akwụkwọ, ebe nlebara anya na-ejikọta okwu ajụjụ na akụkụ nke nwere azịza ya.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Attention Mechanisms quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nleba anya windo slide
Ajụjụ a na-ajụkarị
What is Attention Mechanisms?
Nlebara anya na-eme ka ihe nlere anya kpebie okwu ndị ọzọ dị n'ahịrịokwu kacha karịa mgbe a na-atụgharị okwu ọ bụla. Ọ bụ isi echiche mere transformer - ya mere AI ọgbara ọhụrụ dị ka ChatGPT - kwere omume.
N'otu ahịrịokwu, gịnị ka usoro nlebara anya na-eme?
Nlebara anya na-agbakọ nha nke na-ekpebi ego ole akara nke ọ bụla kwesịrị ịdọrọ na akara ndị ọzọ mgbe ị na-emepụta ihe nnọchianya ya.
Kedu akwụkwọ akara ngosi 2017 webatara ihe owuwu nke transformer?
'Ntị bụ ihe niile ị chọrọ' (Vaswani et al., 2017) tụrụ aro ka transformer, nke na-adabere na nlebara anya kama nlọghachite.
Kedu ihe a na-agbakọ vector atọ maka nlebara anya nke ọ bụla?
Token ọ bụla na-ewepụta ajụjụ, igodo, na uru; dakọtara ajụjụ na igodo iji tụọ ụkpụrụ.
Na usoro softmax (QK^T / √d_k) V, gịnị kpatara kewaa site na √d_k?
Ịtụgharị site na mgbọrọgwụ square nke akụkụ igodo ahụ na-egbochi nnukwu ngwaahịa ntụpọ ga-eme ka softmax banye n'ime obere gradients, na-eme ka ọzụzụ kwụsie ike.
Kedu ihe kpatara nlebara anya onwe onye ọkọlọtọ ji adị ọnụ maka ntinye ogologo ogologo?
Ihe nrịbama ọ bụla na-aga na akara ngosi ọ bụla ọzọ, yabụ ọnụọgụ atụnyere nha dị ka n², na-eme ogologo usoro ogologo oge na ebe nchekwa.