Laajte buñ boole
Grouped-Query Attention (GQA) anam la wu ñuy wàññi memory biñ soxla ci defar mbind, ci bàyyi boppu laaj yu bari ñu bokk benn caabi ak benn boppu valeur.
Résumé
It makes large models much faster to serve with almost no quality loss.
Plongeur bu xóot
Ci layer attention multi-head buñ miin, bopp bu nekk amna ay laaj, ay caabi ak ay valeur. Bu ñuy génne, caabi yi ak valeur yi ci token yi njëkk yépp dañu leen denc ci cache ('cache KV') suko defee model bi baña leen xaymaat. Ak bopp yu bari ak contexte yu gudd, cache bi dafay nekk lu rëy ba noppi di ëpp doole ci yaatuwaayu mémoire bi ci diiru inference. GQA, Google gëstukat yi ñoo ko dugal ci 2023, dafay boole boppu laaj yi ba noppi jox mbooloo mu nekk benn boppu caabi ak valeur yuñ bokk. Soo amee 32 boppu laajte waaye 8 grupu KV kese, cache KV dafay wàññeeku lu tollu ci ñeenti yoon. Lii mingi toog ci diggante xel mu bari-bopp (bopp bu nekk dafa tàqaloo) ak xel mu-laaj (benn KV buñ bokk ngir bopp yépp), jàpp gaawaayu MQA bi gëna bari, boole ci tëye kalite bi jege bàyyi xel bu mat. Llama 2 70B ak yeneen model yu ci topp dañu ko jëfandikoo.
Gis-gis xarala
Kalite bàyyi xel mingi aju ci am yoon yu bari yu wuute ci laaj, waaye dafay nangu ñu séddoo caabi yi ak valeur yi. GQA dafay jëfandikoo asymétrie bii: dafay tëye bépp boppu laaj waaye dafay nuru bépp boppu KV buñ bokk ci laaj yi ci grupu bi. Sakkanal bi dafay ñëw ci inference, fu cache KV mooy konsomatër bi gëna mag ci yaatuwaayu mémoire bi; boppu KV yu néew dafay tekki done yu néew yuñ wara jàng ci token buñ defar. Model yi dañuy faral di 'uptrained' ci diir bu gàtt ngir soppi benn checkpoint bu am bopp yu bari ci benn GQA.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu bàyyi xel ci laaj yuñ boole
GQA leegi default standard la ci model yu ubbeeku yi ndax dafay jaay njëgu kalite bu tuuti ngir am ndam yu mag. Xaarandil mu gëna boole ak yeneen pexe yu am njariñ yu melni FlashAttention, KV-cache quantization, ak ay pexe yu bees yu melni bàyyi xel bu nëbbu bu bari bopp buy gëna kompresse cache bi. Lu palanteer yi di màgg, di saytu dayo cache KV dina des ci jafe-jafe jëmmal, ba noppi séddoo boppu GQA-style dina des ci levier bu am solo.
Doxal ci àdduna dëgg
Llama 2 70B ak Llama 3 jëfandikoo GQA ngir liggéey ci ay mbir yu gudd ak cache KV bu gëna ndaw
Wàññi mémoire GPU suko defee model chat bu mag mëna ànd ak gaawaay yu néew wala yu gëna yomb
Gaawaay defar token-ci-token ci API yi ñuy defaree KV-cache bandwidth mooy jafe-jafe bi
Fexe bamu gëna rëy ngir mëna jëfandikoo jëfandikukat yu bari benn yoon te du yàq mémoire bi
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Grouped-Query Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Laajte yu bari
Laaj yi ñuy faral di laaj
What is Grouped-Query Attention?
Grouped-Query Attention (GQA) anam la wu ñuy wàññi memory biñ soxla ci defar mbind, ci bàyyi boppu laaj yu bari ñu bokk benn caabi ak benn boppu valeur. Dafay tax model yu mag yi gëna gaaw ci liggéey te daanaka du ñàkk benn kalite.
Ci Grouped-Query Attention, lan lañu bokk ci mbooloo mu boppu laaj yi?
GQA dafay tàqale boppu laaj yi waaye dafay may kuréel gu nekk ci ñoom ñu bokk benn boppu caabi ak valeur, di wàññi cache KV.
GQA mooy gëna mëna wax ni diggu suuf la ci diggante ban ñaari extrême?
Bopp bu bari dafay jox bopp bu nekk KV boppam; multi-query bokk benn KV ngir bopp yépp; GQA mingi toog ci digg bi ak ay kuréel KV yu néew.
Ban wàll ci inference la GQA gëna xéewale?
Sakkanal bi dafay wane ci jamonoy defar, fu jàng cache KV mooy njëgu bandwidth memory bi gëna am solo.
Sudee benn model amna 32 boppu laaj ak 8 grupu KV, cache KV bi dafay wàññeeku ci lu tollu ci ban facteur?
32 boppu laajte xaaj 8 grupu KV yuñ bokk dafay joxe lu tollu ci ñeenti yoon wàññeeku ci caabi ak valeur yiñ denc.
Lan moo tax GQA mëna tëye li gëna bari ci kalite model bi doonte dañu bokk boppu KV yi?
Attention dafay muñ séddoo caabi ak valeur yu gëna baax moo gën muñ ñàkka am yoon yu wuute ci laaj, kon GQA dafay tëye kalite bi.