Laajte yu bari
Multi-Query Attention (MQA) dafay sakkanal mémoire ci transformateur biy séddoo benn xeetu caabi ak valeur ci bépp boppu attention.
Résumé
It dramatically speeds up text generation by shrinking the memory the model must shuffle around.
Plongeur bu xóot
Foofu ñuy bàyyi xel ci bopp yu bari dafay jox bopp bu nekk laaj boppam, caabi ak projection valeur. Ci jamonoy generation, caabi yi ak valeur yi ci token yi weesu yépp dañu leen wara cache ak chargewaat ci jéego bu nekk - cache KV bi mooy nekk bottleneck bi gëna mag, ndax jàng ko ci mémoire moo gëna yeex math bi ci boppam. Multi-Query Attention, bi Noam Shazeer xalaat ci 2019, dafay tëye ay projection laaj yu wuute ci bopp bu nekk waaye dafay dindi caabi yi ak valeur yi ci benn bopp buñ bokk. Loolu dafay wàññi cache KV ci lu tollu ci limu bopp yi, yenn saa mu gëna ndaw 8x ba 64x. Lépp soo ko boolee mooy decodage autorégressive bu gëna gaaw ak emprent mémoire bu gëna woyof, ak tuuti ci kalite bi. Diggu suuf, Grouped-Query Attention, mooy ekilibre kompromis bi.
Gis-gis xarala
Ci MQA, diisaayu laajte yi ba leegi dañuy defar H vecteur laajte yu wuute, waaye benn projection key ak benn projection valeur lañu bokk ci bopp yépp. Bopp bu nekk dafay xayma attention bi ci laaj boppam ci benn caabi ak valeur yi. Ndax tensor K ak V yiñ cache amatu ñu benn scale ak limu bopp yi, bandwidth memory bi ci decodage bi dafay wàññeeku bu baax - te bandwidth, nekkul calcul, mooy gaawaayu defar buntu yi ci accelerateur yu bees yi.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu bàyyi xel ci laaj yu bari
MQA dafa wane ni mën nga dagg boppu caabi / valeur yu bari te du am benn loraange, te gis-gis boobu leegi dafay forme LLM bu gaaw bu nekk. Doxalin bi dafa gëna jëm ci Grouped-Query Attention (GQA), ñu koy jëfandikoo ci Llama 2/3 ak ñeneen ñu bari, ñuy jëfandikoo ay grupu KV yu néew, du benn ngir defaraat kalite bi, di tëye li gëna bari ci gaawaay bi. Liggéey bi ci kanam dafay boole xalaat yooyu ak KV-cache compression, quantisation, ak bàyyi xel ci lu bari lu nëbbu ngir push contexte yu gëna gudd ak serwiis bu gëna xéewale.
Doxal ci àdduna dëgg
Gaawaay defar token-ci-token ci assistant chat fu cache KV, du ordinatër bu ñor, di tënk produit bi.
Google's PaLM, bi jëfandikoo Fexe Laajte yu bari ngir mëna am njariñ ci anam wu yaatu.
Dimbali jëfandikukat yu bari ci benn GPU ci wàññi memory cache KV bu laaj bu nekk.
Groupe-Query Fexe ci Llama 2 70B ak Llama 3, ab mbokk buy yemale gaawaayu MQA ak kalite bàyyi xel bu mat sëkk.
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Multi-Query Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Laajte buñ boole
Laaj yi ñuy faral di laaj
What is Multi-Query Attention?
Multi-Query Attention (MQA) dafay sakkanal mémoire ci transformateur biy séddoo benn xeetu caabi ak valeur ci bépp boppu attention. Dafay gaawlu bu baax defar mbind ndax dafay wàññi memory bi model bi wara jaxasoo.
Lan mooy Multi-Query Attention bokk ci bépp boppu bàyyi xel?
MQA dafay tàqale ay laaj ci bopp bu nekk waaye dafay bokk benn projection bu am solo ak benn projection valeur ci bopp yépp.
Lan mooy jafe-jafe bi njëkk bi MQA di saafara ci jamonoy defar autoregressive?
Yarsewaat cache KV bu mag bi jéego bu nekk dafay tënku ci yaatuwaayu band bi; MQA dafay wàññi cache bi ngir gaawlu dekodage bi.
Ci ban facteur la MQA di wàññi cache KV bi?
Ginaaw caabi yi ak valeur yi dañuy wàcci joge ci boppu H dem ci benn, cache bi dafay wàññeeku ci limu boppu yi.
Lan mooy njariñ li gëna am solo ci jëfandikoo MQA?
Séddoo caabi yi ak valeur yi dafay wàññi tuuti kàttan representation bi, moo waral kalite bi wàññeeku tuuti ci weccoo gaawaay bu rëy.
Ban anam moo nekk ci digganté bàyyi xel ci bopp yu bari ak MQA?
Grouped-Query Attention (GQA) dafay jëfandikoo ay grupu KV yu bari ci barabu benn, ngir yemale kalite ak gaawaay.