Ntị nlebara anya
FlashAttention bụ algọridim na-arụ ọrụ ebe nchekwa nke na-agbakọ otu nlebara anya dị ka ndị na-agbanwe ọkọlọtọ mana na-enweghị mgbe ọ na-edepụta nnukwu nlebara anya matriks iji belata ebe nchekwa GPU.
Nchịkọta
It made long-context training and inference dramatically faster and cheaper.
Ime miri emi
Nlebara anya ọkọlọtọ na-agbakọ akara maka ụzọ akara ọ bụla, na-emepụta matriks N-by-N. Maka usoro akara ngosi 4,000 nke ahụ bụ akara nde 16, yana matriks ga-ederịrị ma gụọ ya azụ site na ebe nchekwa bandwidth dị elu nke GPU (HBM). Okporo ụzọ ebe nchekwa ahụ, ọ bụghị mgbakọ na mwepụ, bụ ezigbo ihe mgbochi. FlashAttention, nke Tri Dao na ndị ọrụ ibe webatara na 2022, na-ahazigharị mgbakọ ahụ ka matriks ahụ ghara ịdị adị nke ọma. Ọ na-ahazi usoro ahụ na taịlị dabara na obere GPU nke SRAM, na-agbakọ softmax na-abawanye ka ọ na-aga. Nsonaazụ ya bụ mgbakọ na mwepụ na nlebara anya ọkọlọtọ mana ọ na-eji ebe nchekwa dị obere ma na-agba ọsọ ọsọ ọtụtụ ugboro, na-eme ka windo okirikiri dị ogologo ogologo.
Nghọta nka nka
Aghụghọ a bụ 'online softmax' jikọtara ya na tiling. FlashAttention na-ebu obere ajụjụ, igodo, na ụkpụrụ n'ime SRAM, na-agbakọ nsonaazụ nlebara anya nke ọma, wee weghachi ego na-agba ọsọ ka ihe mgbochi ọhụrụ na-abịarute ka softmax normalization na-adịkwa mma n'ahụghị akara niile n'otu oge. N'ihi na ọ dịghị mgbe ọ na-echekwa matrix N-by-N zuru ezu na HBM, ebe nchekwa na-akwụ ụgwọ n'ahịrị ka ọ bụghị nke anọ, na kernel na-ejikọta ya n'ime otu GPU iji belata ebe nchekwa na-agụ ma na-ede.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke FlashNtị
FlashAttention abụrụla ngọngọ ụlọ nke ndabara. FlashAttention-2 kwalitere nkewa ọrụ GPU, yana FlashAttention-3 na-erigbu atụmatụ ngwaike Hopper ọhụrụ dị ka asynchrony na FP8 dị ala. Na-atụ anya na-aga n'ihu na-emewe ya na ibe, ntinye miri emi n'ime sava ntinye akwụkwọ maka ogologo akwụkwọ, yana ụdị dị iche iche echedoro maka nleba anya na windo na-amị amị. Ka windo ndị gbara ya gburugburu na-erute n'ọtụtụ nde akara, IO-maara kernel dị ka nke a ka dị mkpa maka idobe ọzụzụ na inye ụgwọ ọrụ.
Mmejuputa n'ezie n'ụwa
Ọzụzụ nnukwu ụdị asụsụ dị ka Llama na sistemụ ụdị GPT ngwa ngwa yana ọnụ ahịa GPU dị ala
Na-eje ozi ndị enyemaka nkata ogologo okwu na-etinye akwụkwọ ma ọ bụ codebase na-enweghị ebe nchekwa na-agwụ
Na-emewanye pipeline nchịkọta akwụkwọ ngwa ngwa na-ahazi iri puku kwuru iri puku akara n'otu oge
Ike ọhụụ na multimodal transformers ebe ogologo usoro nke patches oyiyi na-eme ka uche dị oke ọnụ
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the FlashAttention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Mgbakwunye ọnọdụ Rotary
Ajụjụ a na-ajụkarị
What is FlashAttention?
FlashAttention bụ algọridim na-arụ ọrụ ebe nchekwa nke na-agbakọ otu nlebara anya dị ka ndị na-agbanwe ọkọlọtọ mana na-enweghị mgbe ọ na-edepụta nnukwu nlebara anya matriks iji belata ebe nchekwa GPU. O mere ka ọzụzụ na ntinye aka ogologo oge dị ngwa ngwa ma dị ọnụ ala.
Kedu ihe bụ isi ọkpọ FlashAttention lekwasịrị anya na nlebara anya ọkọlọtọ?
Nlebara anya ọkọlọtọ na-ejikọta ebe nchekwa: imechi nnukwu matriks N-by-N gaa na site na ebe nchekwa bandwit dị elu na-achịkwa oge, ọ bụghị mgbakọ na mwepụ n'onwe ya.
Kedu ka mmepụta FlashAttention si atụnyere nlebara anya ọkọlọtọ?
FlashAttention na-agbakọ kpọmkwem otu ụkpụrụ nlebara anya; ọ na-ahazigharị mgbakọ na mwepụ ka ọ ghara imezu matriks ahụ.
Kedu ebe nchekwa GPU nke FlashAttention na-erigbu site na nhazi data na taịlị?
FlashAttention na-ebu obere taịlị n'ime GPU's ngwa ngwa on-chip SRAM, na-eme ka ọrụ dị arọ dị nso na nkeji mgbakọ.
Kedu usoro na-eme ka FlashAttention gbakọọ softmax na-ahụghị akara niile n'otu oge?
softmax dị n'ịntanetị na-edobe oke na nchikota na-agba ọsọ, na-eweghachi nsonaazụ akụkụ ka taịlị ọhụrụ na-abịarute ka nhazi ikpeazụ ahụ ziri ezi.
Kedu ihe FlashAttention-3 mere kpọmkwem?
Ejikọtara FlashAttention-3 na Hopper GPU nke ọgbara ọhụrụ, na-eji ogbugbu asynchronous yana FP8 dị ala maka ịgba ọsọ ọzọ.