Flash
FlashAttention ni algorithm ikora neza yibuka igereranya kwitabwaho kimwe na transformateur zisanzwe ariko utarigeze wandika matrike nini yo kwitondera kugabanya GPU.
Incamake
It made long-context training and inference dramatically faster and cheaper.
Kwibira cyane
Icyitonderwa gisanzwe kibara amanota kuri buri jambo ryibimenyetso, bitanga matrike ya N-by-N. Kuri 4000-token ikurikiranye ni amanota miliyoni 16, kandi matrix igomba kwandikwa no gusomwa uhereye kuri GPU yibuka cyane (HBM). Urwo rugendo rwo kwibuka, ntabwo ari imibare, nukuri. FlashAttention, yatangijwe na Tri Dao na bagenzi be mumwaka wa 2022, ivugurura ibarwa kuburyo matrix itigera iba yuzuye. Itunganya urukurikirane muri tile ihuye na GPU ntoya, ultra-yihuta kuri chip SRAM, kubara softmax buhoro buhoro uko igenda. Igisubizo ni imibare ihuye nibitekerezo bisanzwe ariko ikoresha ububiko buke cyane kandi ikora inshuro nyinshi byihuse, itanga Windows ndende cyane.
Ubushishozi
Amayeri ni 'kumurongo wa softmax' uhujwe na tiling. FlashAttention yikoreza uduce duto twibibazo, urufunguzo, nagaciro muri SRAM, ibara ibisubizo byigice cyibisubizo, kandi ikuraho amafaranga yo gukora nkuko bice bishya bigeze bityo softmax isanzwe igumaho neza utabonye amanota yose icyarimwe. Kuberako itigera ibika matrike yuzuye ya N-by-N muri HBM, umunzani wibukwa umurongo aho kuba inshuro enye, kandi intangiriro ihujwe nigikorwa kimwe cya GPU kugirango igabanye kwibuka buhoro buhoro gusoma no kwandika.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza ka Flash
FlashAttention yahindutse inyubako idasanzwe. FlashAttention-2 yatezimbere GPU igabana, kandi FlashAttention-3 ikoresha ibikoresho bishya bya Hopper ibikoresho nka asynchrony hamwe na FP8 idasobanutse neza. Tegereza gukomeza gufatanya gushushanya hamwe na chip, kwinjiza byimbitse muri seriveri zifatika kubisobanuro birebire, hamwe nibihinduka byahujwe gake cyangwa kunyerera-idirishya. Mugihe windows isunika yerekeza kuri miriyoni yikimenyetso, IO-izi intete nkiyi ikomeza kuba ingenzi kugirango ukomeze amahugurwa no gutanga ibiciro.
Gushyira mu bikorwa Isi
Guhugura indimi nini nka Llama na GPT-yuburyo bwa sisitemu byihuse kandi ku giciro gito cya GPU
Gukora ibiganiro birebire bifasha abafasha gufata ibitabo byose cyangwa codebase utabuze kwibuka
Kwihutisha inyandiko-incamake imiyoboro itunganya ibimenyetso ibihumbi icumi icyarimwe
Imbaraga zicyerekezo hamwe na transformateur ya multimodal aho urutonde rurerure rwibishusho bituma ibitekerezo bihenze
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the FlashAttention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Ibirindiro byumwanya
Ibibazo bikunze kubazwa
What is FlashAttention?
FlashAttention ni algorithm ikora neza yibuka igereranya kwitabwaho kimwe na transformateur zisanzwe ariko utarigeze wandika matrike nini yo kwitondera kugabanya GPU. Cyakoze amahugurwa maremare hamwe no gufata umwanzuro byihuse kandi bihendutse.
Niyihe ntego nyamukuru FlashAttention yibitekerezo mubisanzwe?
Icyitonderwa gisanzwe ni kwibuka-guhuza: guhinduranya materix nini ya N-by-N kugeza no kuva murwego rwohejuru rwibuke rwiganje igihe, ntabwo arithmetic ubwayo.
Nigute umusaruro wa FlashAttention ugereranije no kwitabwaho bisanzwe?
FlashAttention ibara agaciro kamwe ko kwitabwaho; gusa itunganya ibarura kugirango wirinde kwiyerekana matrix yuzuye.
Nibihe bikoresho bya GPU FlashAttention ikoresha mugutunganya amakuru muri tile?
FlashAttention yapakira udupapuro duto muri GPU yihuta kuri chip SRAM, igakomeza imirimo iremereye hafi ya compte.
Nubuhe buryo butuma FlashAttention ibara softmax utabonye amanota yose icyarimwe?
Kumurongo wa softmax ukomeza gukora ntarengwa hamwe nigiteranyo, ukuraho ibisubizo byigice nkuko amabati mashya ageze bityo ibisanzwe byanyuma nibyo.
Niki FlashAttention-3 yakoresheje muburyo bwihariye?
FlashAttention-3 yateguwe hamwe na Hopper GPU igezweho, ikoresheje imikorere idahwitse hamwe na FP8 idasobanutse neza kugirango yihute.