I-FlashAttention
I-FlashAttention i-algorithm enenkumbulo esebenza kahle ehlanganisa ukunaka okufanayo ncamashi nama-transformer ajwayelekile kodwa ngaphandle kokubhala i-matrix enkulu yokunaka ukuze ibambezele inkumbulo ye-GPU.
Uhlolojikelele
It made long-context training and inference dramatically faster and cheaper.
I-Deep Dive
Ukunaka okujwayelekile kubala amaphuzu epheya ngalinye lamathokheni, okukhiqiza i-matrix ye-N-by-N. Ngokulandelana kwamathokheni angu-4,000 lokho amaphuzu ayizigidi ezingu-16, futhi i-matrix kufanele ibhalwe futhi ifundwe kusukela kumemori yomkhawulokudonsa ophezulu we-GPU (HBM). Leyo datha yenkumbulo, hhayi izibalo, iyisisekelo sangempela. I-FlashAttention, eyethulwe ngu-Tri Dao nozakwabo ngo-2022, ihlela kabusha ukubala ukuze i-matrix ingalokothi yenziwe ngokuphelele. Icubungula ukulandelana kwamathayili alingana ku-GPU encane, eshesha kakhulu ku-chip SRAM, ihlanganisa i-softmax ngokukhuphukayo njengoba ihamba. Umphumela uyafana ngokwezibalo nokunaka okujwayelekile kodwa isebenzisa inkumbulo encane kakhulu futhi isebenza ngokushesha izikhathi ezimbalwa, inika amandla umongo omude kakhulu windows.
I-Technical Insight
Iqhinga 'i-softmax eku-inthanethi' ehlanganiswe nokuthayela. I-FlashAttention ilayisha amabhulokhi amancane emibuzo, okhiye, namanani ku-SRAM, ibala ukuphuma kokunaka okuncane, futhi ilinganise ngezibalo ezigijima njengoba amabhulokhi amasha efika ukuze i-softmax normalization ihlale ilungile ngaphandle kokubona wonke amaphuzu ngesikhathi esisodwa. Because it never stores the full N-by-N matrix in HBM, memory scales linearly rather than quadratically, and the kernel is fused into a single GPU operation to minimize slow memory reads and writes.
I-Strategic Impact
Isivinini nesikali
Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.
Finyelela futhi ufinyelele
Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.
Izinqumo ezicacile
Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.
Ikusasa le-FlashAttention
I-FlashAttention isiphenduke ibhulokhi yokwakha ezenzakalelayo. I-FlashAttention-2 ethuthukisiwe ukwahlukanisa umsebenzi we-GPU, futhi i-FlashAttention-3 isebenzisa izici ezintsha zehadiwe ye-Hopper njenge-asynchrony ne-FP8 enembayo ephansi. Lindela idizayini eqhubekayo ehambisana nama-chips, ukuhlanganiswa okujulile kumaseva e-inference kumadokhumenti amade, kanye nokwahluka okushunwe ukunakwa okungatheni noma kwewindi elislayidayo. Njengoba umongo amawindi ephushela ezigidini zamathokheni, izinhlamvu ze-IO-aware ezifana nalezi zihlala zibalulekile ekugcineni ukuqeqeshwa kanye nezindleko zokuhlinzeka zilawuleka.
Ukuqaliswa Komhlaba Wangempela
Ukuqeqesha amamodeli amakhulu olimi afana nesistimu yesitayela se-Llama ne-GPT ngokushesha nangezindleko eziphansi ze-GPU
Inikeza abasizi bengxoxo yomongo omude abangenisa izincwadi zonke noma izisekelo zekhodi ngaphandle kokuphelelwa yinkumbulo
Ukusheshisa amapayipi okufingqa amadokhumenti acubungula amashumi ezinkulungwane zamathokheni ngesikhathi esisodwa
Ukubona amandla kanye nama-multimodal transformer lapho ukulandelana okude kwamapeshi esithombe kwenza ukunaka kubize kakhulu
Izingozi & Guardrails
Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.
Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.
Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.
Ukuqalisa Umhlahlandlela
Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.
Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.
Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.
Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the FlashAttention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Rotary Position Embeddings
Imibuzo evame ukubuzwa
What is FlashAttention?
I-FlashAttention i-algorithm enenkumbulo esebenza kahle ehlanganisa ukunaka okufanayo ncamashi nama-transformer ajwayelekile kodwa ngaphandle kokubhala i-matrix enkulu yokunaka ukuze ibambezele inkumbulo ye-GPU. Kwenze ukuqeqeshwa komongo omude nokuchazwa kusheshe kakhulu futhi kushibhe.
Ithini ibhodlela elikhulu le-FlashAttention eqondiwe ekunakeni okujwayelekile?
Ukunaka okujwayelekile kuboshwe kwinkumbulo: ukuvala i-matrix enkulu ye-N-by-N ukuya nokusuka kumemori yomkhawulokudonsa ophezulu kulawula isikhathi, hhayi izibalo ngokwazo.
Ngabe okukhiphayo kwe-FlashAttention kuqhathaniswa kanjani nokunakwa okujwayelekile?
I-FlashAttention ihlanganisa amanani okunaka afana nse; ivele ihlele kabusha ukubala ukuze kugwenywe ukwenza i-matrix egcwele.
Iyiphi inkumbulo ye-GPU i-FlashAttention eyisebenzisayo ngokucubungula idatha kumathayili?
I-FlashAttention ilayisha amathayela amancane ku-GPU ye-chip esheshayo ye-SRAM, egcina umsebenzi osindayo useduze namayunithi wokubala.
Iyiphi inqubo evumela i-FlashAttention ihlanganise i-softmax ngaphandle kokubona wonke amaphuzu ngesikhathi esisodwa?
I-softmax eku-inthanethi igcina ukusebenza kwe-maximum nezibalo, inciphisa imiphumela eyingxenye njengoba kufika amathayela amasha ukuze ukujwayela kokugcina kulunge.
I-FlashAttention-3 isebenzise ini ngokukhethekile?
I-FlashAttention-3 yaklanywa ngokuhlanganyela nama-GPU e-Hopper esimanje, kusetshenziswa ukusebenza okuvumelanayo kanye ne-FP8 enembayo ephansi ukuze kuthuthukiswe isivinini.