Làkk AI GUIDE
Cache KV
Cache KV dafay denc caabi ak vecteur valeur yi transformateur bi jota xayma ci token yu njëkk yi, kon jarul ñu xaymaat leen ci baat bu bees bumu defar.
Déglul tegtal biiIA-nettali · ~2 min
Ci xët wii2 simili jàng
Résumé
Mooy sabab bi gëna mag bi tax defar mbind gaaw - te li gëna am solo mooy lekk sa mémoire GPU ci waxtaan yu yàgg.
Plongeur bu xóot
Transformatër yi dañuy defar benn token benn yoon, te token bu bees bu nekk dafa wara méngoo ak token bu njëkk bi. Mekanismu bàyyi xel bi dafay soppi token bu nekk ci laaj, caabi ak vecteur valeur. Bu cache amul, defar nimero token 1,000 dafay tekki ni dañuy xaymawaat caabi yi ak valeur yi ngir 999 token yi njëkka am ci jéego bu nekk - liggéey bu kwadratik, bu yàqu. Cache KV dafay denc vecteur key ak valeur yooyu ginaaw biñu leen njëkkee xayma ba noppi jëfandikoowaat leen, kon jéego bu bees bu nekk dafay xayma vecteur yi ngir benn token bi gëna bees daal di dem ci cache biñ denc. Loolu dafay wàññi njëgu token bu nekk ci eskalaasioŋ ak guddaayu toppalante ba ci lu tollu ci constant. Kompromis bi mooy mémoire bi: cache bi dafay màgg lineairement ak guddaayu contexte bi, limu layers yi, ak boppu attention yi, lu ci bari mooy nekk consommateur mémoire bi gëna am doole ci sarwiisu contexte bu yàgg.
Doxal ci àdduna dëgg
Gaawaale tontu chatbot ci jëfandikoowaat caabi/valeur yiñ denc ci jaar-jaaru waxtaan wi, moo gën ñu koy defaraat saa yu nekk.
Cache prefix biy séddoo cache bi ci sistem bu yàgg ci jëfandikukat yu bari, wàññi njëg ak latency.
vLLM's PagedAttention dafay yor cache KV ci ay blok ngir mëna def ay laaj yu bari ci benn GPU ci anam wu jaar yoon.
Quantizing cache KV ngir wàññi precision ngir mëna ànd ak contexte yu gëna gudd ci memory GPU bu néew.
Ëlëgu KV Cache
Bi palanteer yi di tàmbalee ci téemeeri junni token, cache KV dafay nekk bottleneck bu mag, kon innovation bi dafa am doole: cache quantization ci 8 wala 4 bits, politiku dàq yuy daaneel token yu amul solo, cross-request prefix sharing, ak offloading ci CPU wala disk. Coppite ci architecture yu melni bàyyi xel ci bopp yu bari dañuy tënk cache bi ci boppam. Xaarandil wéyal co-design ci anam yu ñuy bàyyi xel ak sistemu mémoire yuñ fexe ngir liggéey ci contexte yu gudd lool ci njëg yu yomb ak ci produit yu bari.
Teg ko ci jëf
Lu tax mu am solo
- Gaawaay ak yaatuwaay. Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
- Dugg ak yegg. Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
- dogal yu gëna leer. Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Moytul
- Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
- Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
- Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Jéego yi ci topp
- Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
- Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
- Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
- Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Bulletin bu amul fayda
Jàngal xibaar AI bu bes
Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.
Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.
Nattal sa bopp
Take the KV Cache quiz
Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.
Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax
Laaj yi ñuy faral di laaj
Luy KV Cache?
Cache KV dafay denc caabi ak vecteur valeur yi transformateur bi jota xayma ci token yu njëkk yi, kon jarul ñu xaymaat leen ci baat bu bees bumu defar. Mooy sabab bi gëna mag bi tax defar mbind gaaw - te li gëna am solo mooy lekk sa mémoire GPU ci jamonoy waxtaan yu yàgg.
Luy denc ci cache KV?
Cache KV dafay tëye caabi ak vecteur valeur yu bawoo ci jeton yu njëkk yi suko defee ñu mëna leen jëfandikoowaat ci barabu ordinatër.
Ban jafe-jafe la cache KV di njëkka saafara?
The cache reuses keys and values computed for earlier tokens instead of rebuilding them at every decode step. The new query still attends over cached positions, so attention work and cache memory continue to grow with context length.
Lan mooy njariñu cache KV?
Cache bi dafay màgg ak guddaayi sekans yi, diisaay yi, ak bopp yi, di faral di nekk konsomatër bi gëna am solo ci mémoire GPU ci sarwis yu yàgg yi.
Ban pexe mooy wàññi dayo cache KV ci séddoo caabi ak valeur ci boppu attention yi?
Laajte buñ boole ak laajte yu bari dañuy may njiitu laajte yu bari ñu séddoo ensemble caabi/valeur yu néew, di wàññi cache bi bu baax.
Lan mooy ñaari fase yi ci transformateur bi méngoo ak cache KV bi?
Prefill dafay feesal cache bi ak K/V bi ci laaj bi; decode dafay yokk benn K/V token bu bees jéego bu nekk ba noppi di topp cache bi.
Weyal di jàng
Gid yu jëm ci loolu
Tann nañu yeneen njiit ngir topic bii