Flash Fexe
Flash Attention anam bu am xel la ngir xayma jéego yi ci biir Transformers te doo musa bind matrix bu mag biy yeexal mémoire bi.
Résumé
It makes long-context models far faster and more memory-efficient without changing their math.
Plongeur bu xóot
Fàttaliku buñ miin dafay méngale token bu nekk ak beneen token bu nekk, mu defar matrix poñ N-by-N buy màgg quadratically ak guddaayu toppalante bi. Ci anam wu amul benn xam-xam, matrix boobu dañu ko bind ak lire ci GPU memory bandwidth bu kawe (HBM), te shuttling bi - du yokk - mooy bottleneck dëgg. Flash Attention, bi Tri Dao ak ay naataango dugal ci 2022, dafay yamale xayma bi suko defee matrix bi du musa denc lépp. Dafay liggéey ci laaj yi, caabi yi, ak valeur yi ci ay carreaux yu ndaw yu méngoo ak SRAM bu gaaw bi ci puce bi, xayma ay resultaa yu néew, ba noppi boole leen ci jëfandikoo benn trick running-softmax ci net bi. Ci wàllu math, génne gi dafay nuru ak bàyyi xel bu bari waaye dafay jëfandikoo mémoire lineaire te dafay daw lu bari yoon, rawatina ci sequence yu gudd.
Gis-gis xarala
Li gëna am solo mooy tiling boole ci softmax ci net bi. Softmax dafay soxla rang poñ yépp ngir mëna xayma limuy denominateur, waaye Flash Attention dafay wéy di am maximum buy daw ak sum buy daw ndax dafay stream carreau bu nekk, di rescaling génne yu njëkk yi suko defee resultaa bi mujj nekk gëna jub. Ndax poñ yi ci digg yi dañuy des ci SRAM (range yu magnitude yu gëna gaaw ci HBM), algorithm bi IO-aware la: dafay wàññi li ñuy jàng ak bind ci mémoire bi moo gën jëfandikoo arithmétique bu ñor.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu Flash Attention
Flash Attention nekkaatna luñuy tabax, ak FlashAttention-2 ak FlashAttention-3 di gëna xëcc ay GPU yu bees yu melni H100 ci gëna mëna xaaj liggéey bi ak di jëfandikoo yooni FP8 yu woyof. Xaarandil wéyal co-design ak hardware, boole bu gëna dëgër ci tàggat ak kaadar inference, ak variants yuñ defar ngir sparse, palanteer buy gliise, ak bàyyi xel ci contexte bu gudd lool. Ginaaw palanteer yi dañuy dem ba ci ay milioŋ ciy token, kernel yu IO-aware yu mel ni yii dañuy wéy di am solo ngir mëna tëye mémoire ak gaawaay.
Doxal ci àdduna dëgg
Taggat xeetu làkk yu mag yu melni Llama ak sistem GPT-class ak palanteer yu gëna gudd ci njëgu mémoire bu gëna néew.
Li gëna gaaw ci jàppale assistant chat yi ci gaawal etape prefill bi ñuy njëkka lire ab laaj bu gudd.
Fexe ba jumtukaayi jàngat këyit yi mëna jël téere yi yépp wala base code yi ci fexe ba ñu mëna bàyyi xel ci toppalante yu gudd yi ci benn GPU.
Transformatër yiy dundal gis-gis ak déglu, fu ay dugal yu am resolusioŋ bu kawe di defar ay token yu gudd lool.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Flash Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Fexe Rollout ak Dagg Bopp
Laaj yi ñuy faral di laaj
What is Flash Attention?
Flash Attention anam bu am xel la ngir xayma jéego yi ci biir Transformers te doo musa bind matrix bu mag biy yeexal mémoire bi. Dafay tax model yu am contexte yu gudd yi gëna gaaw, gëna am memory bu baax te du soppi seen math.
Lan mooy jafe-jafe bi gëna mag bi Flash Attention di wut?
Flash Attention xamna IO: dafay wàññi done yi ñuy yóbbu ci diggante SRAM bu gaaw bi ci puce bi ak mémoire bu yeex bi am bandwidth bu kawe, te loolu mooy bottleneck bi dëgg, te du arithmétique bi ci boppam.
naka la Flash Attention di moytu denc matrix N-by-N bu mat sëkk?
Dafay tile xayma bi suko defee bloc bu nekk mëna ànd ak SRAM bu gaaw, xayma ak dajale ay génnekaay yu néew te du musa materialisee matrix bi yépp ci HBM.
Ban pexe mooy may Flash Attention mu xayma softmax ci anam wu jaar yoon te du gis liiñ bi yépp benn yoon?
Softmax bi ci net bi dafay tëye ab maksimum buy daw ak ab denominatër buy daw bi muy streaming ay carreaux, di rescaling génnekaay yu njëkk yi suko defee normalisation bu mujj bi gëna jubal.
Lan moo waral Flash Attention di gëna am njariñ ci toppalante yu gudd yi?
Matrix attention naïf bi dafay nekk N-carré ci mémoire bi, kon moytu dencukaay bi yépp mooy sakkanal xaalis bi gëna mag sudee sequence yi dañu gudd.
Lan mooy 'IO-xam' jëmmal Flash Attention njëkka wàññi?
IO-aware dafay tekki ni algorithm bi dañu ko tëral ci njëgu toxal done yi ci hierarchie mémoire bi, wàññi trafik HBM moo gën jëfandikoo arithmetik.