Fiiri fiiro gaar ah
FlashAttention waa algorithm-ka wax-ku-oolka ah ee xusuusta kaas oo xisaabiya isla dareenka saxda ah ee beddelayaasha caadiga ah laakiin aan waligiis qorin jaantuska dareenka weyn si loo yareeyo xusuusta GPU-da.
Dulmar
It made long-context training and inference dramatically faster and cheaper.
quusid qoto dheer
Dareenka caadiga ah wuxuu xisaabiyaa dhibcaha lammaane kasta, isagoo soo saaraya jaantuska N-by-N. Isku xigxiga 4,000-token kaas oo ah 16 milyan buundooyinka, iyo shaxanka waa in lagu qoraa oo dib looga akhriyaa xusuusta-bandwidth-sare ee GPU (HBM). Taraafikada xusuusta, ee maaha xisaabta, waa caqabadda dhabta ah. FlashAttention, oo ay soo bandhigeen Tri Dao iyo asxaabtiisa sanadka 2022, ayaa dib u habayn ku sameeya xisaabinta si shaxdu waligiis si buuxda loo hirgelin. Waxay ku socodsiisaa isku xigxiga ee tiles ee ku habboon GPU-ga yar, aadka u degdegga ah ee chip-ka SRAM, xisaabinta softmax si tartiib tartiib ah sida ay u socoto. Natiijadu waxay xisaab ahaan la mid tahay dareenka caadiga ah, laakiin waxay isticmaashaa xusuusta aad uga yar waxayna u socotaa dhowr jeer si dhakhso ah, taasoo awood u siinaysa daaqadaha macnaha guud.
Aragtida Farsamada
Khiyaamadu waa 'softmax online' oo lagu daray tiling. FlashAttention waxay ku shubtaa qaybo yaryar oo su'aalo ah, furayaal, iyo qiyam ah SRAM, waxay xisaabisaa qayb ka mid ah wax soo saarka dareenka, oo waxay dib u xisaabisaa wadarta socodsiinta marka baloogyada cusubi yimaadaan si caadiga softmax ay u ahaato mid sax ah iyada oo aan la arkin dhammaan buundooyinka hal mar. Sababtoo ah waligiis kuma kaydiso matrixka N-by-N ee buuxa ee HBM, xusuusta ayaa si toos ah u miisaamaysa halkii ay ka ahaan lahayd afar-geesood, iyo kernel-ku waxa lagu dhex milmay hal hawlgal oo GPU ah si loo yareeyo akhriska iyo qorista xusuusta gaabis ah.
Saamaynta Istiraatijiyadeed
Xawaaraha iyo miisaanka
Socodka shaqada luqaddu si dhakhso leh ayay u socon kartaa iyada oo aan la hurayn joogteynta.
Helitaanka iyo gaarsiinta
Waxay balaadhisaa gelitaanka luqadaha iyo qaababka isgaarsiinta.
Go'aamo cad
Kooxuhu waxay waqti badan ku qaadan karaan xukunka halka otomaatiggu uu qabanayo ku celcelinta.
Mustaqbalka Fiirinta
FlashAttention wuxuu noqday dhisme aan caadi ahayn. FlashAttention-2 waxay wanaajisay qaybinta shaqada GPU, iyo FlashAttention-3 waxay ka faa'iidaysataa sifooyinka qalabka Hopper ee cusub sida asynchrony iyo saxsan hoose ee FP8. Filo naqshadaynta sii socota ee chips-ka, is dhexgalka qoto dheer ee server-yada macluumaadka ee dukumeentiyada dheer, iyo kala duwanaanshiyaha loo habeeyey dareenka daaqadaha ama simbiriirixan. Maaddaama daaqadaha macnaha guud ay u riixayaan malaayiin calaamado ah, IO-ware kernels sidan oo kale ah ayaa weli muhiim u ah ilaalinta tababarka iyo u adeegida kharashyada la maarayn karo.
Dhaqangelinta Adduunka-dhabta ah
Tababarka moodooyinka luqadaha waaweyn sida Llama iyo nidaamyada GPT-ga si dhakhso leh oo kharash yar GPU ah
U adeegida kaaliyeyaasha wada sheekaysiga muddada-dheer ee soo geliya dhammaan buugaagta ama saldhigyada code-yada iyada oo aan xusuustu ka dhammaan
Dedejinta dhuumaha dukumentiga-koobitaanka kuwaas oo socodsiiya tobanaan kun oo calaamad ah hal mar
Awooda aragga iyo transformers multimodal halkaas oo taxanaha dhaadheer ee dhejiska sawirku ay dareenka qaali ka dhigaan
Khatarta & Dariiqyada Ilaalada
Xaqiiqooyinka dhalanteed waxay si deggan u geli karaan warbixinnada, taageerada socodka, ama natiijooyinka cilmi-baarista.
Dareenka degdega ahi wuxuu abuuri karaa natiijooyin aan iswaafaqayn codsiyada la midka ah.
Xogta qoraalka xasaasiga ah ayaa laga yaabaa in la kashifo haddii kontaroolada gelitaanka ay daciif yihiin.
Qorshe Hawleedka Dhaqangelinta
Qeex qaabka wax soo saarka, codka, iyo heerarka tayada ka hor inta aan la baahin.
Jawaabaha salka ku haya ilo lagu kalsoon yahay mar kasta oo saxnidu ay muhiim tahay.
Hayso isbaarada dib u eegista bini aadamka ee wax soo saarka sare.
Lasoco qaababka guuldarada oo dib u leyli dardargelinta ama socodka shaqada si joogto ah.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the FlashAttention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
Isku-xidhka Booska Wareega
Su'aalaha soo noqnoqda
What is FlashAttention?
FlashAttention waa algorithm-ka wax-ku-oolka ah ee xusuusta kaas oo xisaabiya isla dareenka saxda ah ee beddelayaasha caadiga ah laakiin aan waligiis qorin jaantuska dareenka weyn si loo yareeyo xusuusta GPU-da. Waxay ka dhigtay tababbarka mawduuca dheer iyo ka-fiirsashada si aad u dhakhso badan oo jaban.
Waa maxay cilad-xumada ugu weyn ee FlashAttention bartilmaameedyada dareenka caadiga ah?
Feejignaanta caadiga ahi waxay ku xidhan tahay xusuusta: xidhidhaynta shaxanka weyn ee N-by-N una gudubta xusuusta-bandwidth-sare ayaa xukuma wakhtiga, ma aha xisaabta lafteeda.
Sidee wax-soo-saarka FlashAttention barbardhigaan dareenka caadiga ah?
FlashAttention wuxuu xisaabiyaa qiyamka dareenka saxda ah; waxa ay dib u habaynaysaa xisaabinta si ay uga fogaato in la hirgeliyo matrixka buuxa.
Waa kuwee xusuusta GPU-da ay FlashAttention ka faa'iidaysato iyada oo ku hawlanaysa xogta taayirada?
FlashAttention waxay ku shubtaa dusiyo yaryar oo ku jira GPU-da degdega ah ee-chip-ka SRAM, iyada oo shaqada culus ku haysa meel u dhow unugyada xisaabinta.
Farsamo noocee ah u ogolaanaya FlashAttention xisaabinta softmax iyada oo aan la arkin dhammaan buundooyinka hal mar?
Software-ka online-ka ah ee softmax wuxuu ilaaliyaa socodsiinta ugu badnaan iyo wadarta, dib u habeynta natiijooyinka qayb ahaan marka tiirarka cusubi yimaadaan si caadi ahaan ugu dambeeya ay sax tahay.
Maxay FlashAttention-3 si gaar ah uga faa'iidaysatay?
FlashAttention-3 waxaa la nashqadeeyay Hopper GPU-yada casriga ah, iyadoo la adeegsanayo fulin aan la wada soconin iyo FP8 oo sax ah oo hooseeya ee xawaaraha dheeraadka ah.