I-Flash Attention
I-Flash Attention iyindlela ehlakaniphile yokubala isinyathelo sokunaka ngaphakathi kwe-Transformers ngaphandle kokubhala i-matrix enkulu yokunaka ukuze ubambezele inkumbulo.
Uhlolojikelele
It makes long-context models far faster and more memory-efficient without changing their math.
I-Deep Dive
Ukunaka okujwayelekile kuqhathanisa yonke ithokheni nawo wonke amanye amathokheni, okukhiqiza i-matrix yesikolo se-N-by-N ekhula ngokuphindwe kane ngobude bokulandelana. Ngokungazi, leyo matrix ibhalelwe futhi ifundwe emuva kumemori yomkhawulokudonsa ophezulu we-GPU (HBM), futhi lokho kuvalwa - hhayi ukuphindaphindeka - kuyibhodlela langempela. I-Flash Attention, eyethulwe u-Tri Dao nozakwabo ngo-2022, ihlela kabusha ukubala ukuze i-matrix ingalokothi igcinwe ngokugcwele. Icubungula imibuzo, okhiye, kanye namanani kuthayela amancane alingana ku-chip esheshayo ye-SRAM, ihlanganisa imiphumela engaphelele, futhi iwahlanganise ndawonye kusetshenziswa iqhinga le-inthanethi eligijima-softmax. Okukhiphayo kuyafana ngokwezibalo nokunaka okuvamile kodwa kusebenzisa inkumbulo yomugqa futhi kusebenza ngokushesha izikhathi ezimbalwa, ikakhulukazi ekulandeleni okude.
I-Technical Insight
Iqhinga elibalulekile ukufaka amathayela kanye ne-softmax eku-inthanethi. I-Softmax ivamise ukudinga umugqa wonke wezikolo ukuze ibale inani eliphansi, kodwa i-Flash Attention igcina isamba esiphezulu nesisebenzayo njengoba isakaza ithayela ngalinye, ikala kabusha ingxenye yangaphambili yokuphumayo ukuze umphumela wokugcina ube unembile. Ngenxa yokuthi amaphuzu amaphakathi ahlala ku-SRAM (ama-oda obukhulu ashesha kakhulu kune-HBM), i-algorithm iyazi nge-IO: inciphisa ukufundwa kwememori futhi ibhale kunemisebenzi ye-arithmetic eluhlaza.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Le-Flash Attention
I-Flash Attention isiphenduke ibhulokhi yokwakha ezenzakalelayo, i-FlashAttention-2 ne-FlashAttention-3 ecindezela ukuphuma okuningi kuma-GPU amasha njenge-H100 ngokuthuthukisa ukuhlukanisa umsebenzi nokusebenzisa izindlela ezinembayo eziphansi ze-FP8. Lindela idizayini eqhubekayo enezingxenyekazi zekhompuyutha, ukuhlanganiswa okuqinile ohlakeni lokuqeqeshwa kanye nezinkomba, kanye nokwahluka okushuniwe ukunaka okuncane, iwindi elislayidayo, kanye nokunaka kokuqukethwe okude kakhulu. Njengoba amawindi womongo anwebeka afinyelela ezigidini zamathokheni, izinhlamvu ze-IO-aware ezifana nalezi zihlala zibalulekile ekugcineni inkumbulo nesivinini sisebenza.
Ukuqaliswa Komhlaba Wangempela
Ukuqeqesha amamodeli olimi amakhulu njenge-Llama nezinhlelo zesigaba se-GPT ezinomongo omude amawindi ngezindleko eziphansi zememori.
Ukunikeza abasizi bengxoxo ngokushesha ngokusheshisa isigaba sokugcwalisa kuqala lapho ukwaziswa okude kufundwa kuqala.
Ukunika amandla amathuluzi okuhlaziya amadokhumenti angenisa izincwadi zonke noma izisekelo zekhodi ngokwenza ukunaka okulandelanayo kube nokwenzeka ku-GPU eyodwa.
Amandla okubona kanye nama-Audio Transformers lapho okokufaka okunokulungiswa okuphezulu kudala ukulandelana kwamathokheni amade kakhulu.
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Flash Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
Ukunakwa Kokukhishwa kanye Nokuthena Ikhanda
Imibuzo evame ukubuzwa
What is Flash Attention?
I-Flash Attention iyindlela ehlakaniphile yokubala isinyathelo sokunaka ngaphakathi kwe-Transformers ngaphandle kokubhala i-matrix enkulu yokunaka ukuze ubambezele inkumbulo. Kwenza amamodeli womongo omude asheshe kakhulu futhi asebenzise inkumbulo kahle ngaphandle kokushintsha izibalo zawo.
Iliphi ibhodlela eliyinhloko i-Flash Attention eliqondisayo?
I-Flash Attention i-IO-aware: yehlisa idatha evaleka phakathi kwe-SRAM esheshayo eku-chip kanye nenkumbulo ehamba kancane yomkhawulokudonsa ophezulu, okuyibhodlela langempela kune-arithmetic ngokwayo.
I-Flash Attention ikugwema kanjani ukugcina i-matrix yokunaka ye-N-by-N ephelele?
Ifaka amathayela ukubala ukuze ibhulokhi ngayinye ilingane ku-SRAM esheshayo, ikhompuyutha futhi iqongelela imiphumela engaphelele ngaphandle kokwenza yonke i-matrix ku-HBM.
Iyiphi inqubo evumela i-Flash Attention ihlanganise i-softmax ngendlela efanele ngaphandle kokubona wonke umugqa ngesikhathi esisodwa?
I-softmax eku-inthanethi igcina inani eliphezulu elisebenzayo nelisebenzayo ngenkathi isakaza amathayela, ikala kabusha okuphumayo okuyingxenye yangaphambili ukuze ukujwayela kokugcina kube ncamashi.
Kungani i-Flash Attention isiza kakhulu ngokulandelana okude?
I-matrix yokunakwa kwe-naive ikala njengo-N-squared enkumbulweni, ngakho ukugwema isitoreji sayo esigcwele kuveza ukonga okukhulu kakhulu lapho ukulandelana kukude.
Yini idizayini ye-'IO-aware' ye-Flash Attention ikubeka kuqala ukuncishiswa?
I-IO-aware isho ukuthi i-algorithm yakhelwe ngokulingana nezindleko zokuhambisa idatha kuhlelo lwenkumbulo, ukunciphisa ithrafikhi ye-HBM esikhundleni sokusebenza kwe-arithmetic.