Dañuy bàyyi xel ci anam yi
Fàttaliku dafay tax model bi tànn ban yeneen baat ci frase bi moo gëna am solo sooy tekki baat bu nekk.
Résumé
It is the core idea that made the transformer — and therefore modern AI like ChatGPT — possible.
Plongeur bu xóot
Attention dafay tontu laaj bu yomb ci baat bu nekk: ban yeneen baat laa wara xoolaat ngir xam lii? Ci atum 2017, këyitu 'Fàttaliku mooy li nga soxla' bi Vaswani ak ay naataango ci Google dugal transformatër bi, biy jëfandikoo bàyyi xel ni motëram bu mag ba noppi bàyyi jëmmal yu yàgg yi. Jeton bu nekk dañu koy soppi def ko ñatti vecteur: ab laaj (lu may seet?), ab caabi (lu may joxe?), ak ab valeur (xibaar bi may yóbbaale). Laajteb benn token dañu koy méngale ak caabi token bu nekk ngir am poid attention, suko defee ñu boole valeur yi. Self-attention dafay def loolu ci benn yoon, suko defee baat bu nekk mëna bàyyi xel ci beneen baat bi. Multi-head attention dafay def ay méngale yu bari yu mel noonu, bu nekk di xoole ci motif yu wuute.
Gis-gis xarala
Math bi dafay am produit tombu: softmax(QK^T / √d_k) V. Produit tombu laajte yi ak caabi yi dañuy wane njariñu peer bu nekk; xaaj ak rasin kare bu dimension clé (√d_k) dafay tere poñ yooyu di gëna rëy; softmax daf leen di def ay poid yuy boole benn; ak yokk ci V dafay defar njaxasu valeur yuñ pondéree. Ndax token bu nekk dafay méngoo ak beneen bi, njëg bi dafay màgg ak kaare guddaayu toppalante - O (n2) - moo tax dugal yu gudd yi seer lool te moo tax optimisation yu melni FlashAttention am.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu jumtukaayi bàyyi xel
Foofu lañu wara bàyyi xel, waaye njëgu quadratic bi moo waral gëstu bu xóot bi. FlashAttention moo gëna gaaw ci bàyyi xel ci liggéey bi, ba noppi gëna sakkanal mémoire bi, ndax dafa defaraat ordinatër bi. Jubluwaay yu bees yi bokkuna ci bàyyi xel bu néew ak ligneer, bàyyi xel ci grupu ak laaj yu bari ngir wàññi mémoire bi ci jamonoy generation, ak jëmmal hybrid yuy jaxase bàyyi xel ak xeetu espace-state yu melni Mamba ngir ay dugal yu gudd lool. Xaarandi sistem yu ëlëg ñu baña bàyyi xel ci neexaay bi ci noonu lañuy bënn courbe coût bi suko defee ñu mëna def ay téere yu guddu wala ñu dugal ay këyit yu bari nekk lu ñuy faral di def te yomb.
Doxal ci àdduna dëgg
Tekki làkku masin, di barab bi model bi di bàyyi xel ci baat yiñ tekki suñuy defar baat buñ tekki bu nekk.
Résumé, muy barab buy jàppale model bi mu bàyyi xelam ci santaane yi gëna am solo ci xët wu gudd.
Jàppalekat kode yiy toppatoo li ñu njëkka wax ci variable yi suñuy wax liiñ bi ci topp.
Tontub laaj ci kaw këyit, fu bàyyi xel di boole kàddu laaj yi ak pàcc bi am tontu bi.
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Attention Mechanisms quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Palanteer buy gliise
Laaj yi ñuy faral di laaj
What is Attention Mechanisms?
Fàttaliku dafay tax model bi tànn ban yeneen baat ci frase bi moo gëna am solo sooy tekki baat bu nekk. Mooy xalaat bi tax soppikat bi - ak IA bu bees bu melni ChatGPT - mëna am.
Ci benn frase, lan la mekanismu bàyyi xel di def?
Attention dafay xayma poids yiy wane ni token bu nekk wara jëlee ci yeneen token yi suñuy forme representation bi.
Ban këyit bu am solo ci 2017 moo dugal architecture transformateur bi?
'Fàttaliku mooy li nga soxla' (Vaswani et al., 2017) dafa tàmbale coppite bi, muy wéeru ci bàyyi xel ci barabu baamtu.
Lan mooy ñetti vecteur yiñ xayma ci token bu nekk ci attention?
Token bu nekk dafay defar ab laaj, ab caabi ak ab valeur; laajte yi dañu leen méngale ak caabi yi ngir diisaayu valeur yi.
Ci formul softmax (QK^T / √d_k) V, lu tax nga xaaj ko √d_k?
Eskalaasioŋ ci rasine kare bu dimension bu am solo bi dafay tere produit dot yu mag yiy push softmax ci gradient yu ndaw, di tëye tàggat yaram.
Lan moo waral bàyyi xel ci sa bopp di seer lool ci ay dugal yu yàgg lool?
Bépp token dafay toppatoo bepp token, kon limu méngale yi dafay tollu ci n2, loolu mooy tax toppalante yu gudd yi di seer ci diir bi ak ci fàttaliku bi.