KV Cache
Cache KV dafay denc caabi ak vecteur valeur yi transformateur bi jota xayma ci token yu njëkk yi, kon jarul ñu xaymaat leen ci baat bu bees bumu defar.
Résumé
It is the single biggest reason text generation is fast — and the main thing eating your GPU memory during long conversations.
Plongeur bu xóot
Transformatër yi dañuy defar benn token benn yoon, te token bu bees bu nekk dafa wara méngoo ak token bu njëkk bi. Mekanismu bàyyi xel bi dafay soppi token bu nekk ci laaj, caabi ak vecteur valeur. Bu cache amul, defar nimero token 1,000 dafay tekki ni dañuy xaymawaat caabi yi ak valeur yi ngir 999 token yi njëkka am ci jéego bu nekk - liggéey bu kwadratik, bu yàqu. Cache KV dafay denc vecteur key ak valeur yooyu ginaaw biñu leen njëkkee xayma ba noppi jëfandikoowaat leen, kon jéego bu bees bu nekk dafay xayma vecteur yi ngir benn token bi gëna bees daal di dem ci cache biñ denc. Loolu dafay wàññi njëgu token bu nekk ci eskalaasioŋ ak guddaayu toppalante ba ci lu tollu ci constant. Kompromis bi mooy mémoire bi: cache bi dafay màgg lineairement ak guddaayu contexte bi, limu layers yi, ak boppu attention yi, lu ci bari mooy nekk consommateur mémoire bi gëna am doole ci sarwiisu contexte bu yàgg.
Gis-gis xarala
Ci diiru 'prefill' model bi dafay jëfandikoo prompt bi yépp ba noppi feesal cache bi; ci 'decode' dafay yokk benn token K / V ci jéego bu nekk ba noppi dellu ci. Dayo cache bi dafay am 2 (K ak V) × couche × bopp × bopp_dim × sequence_length × lote, ci njubte giñ tànn. Ngir dakkal loolu, xeetu jamono jii dañuy jëfandikoo laaj-grupp wala laaj-jàngat yu bari ngir séddoo caabi / valeur ci bopp yi, ba noppi sistem yiy liggéey lu melni vLLM dañuy jëfandikoo PagedAttention ngir xaaj cache ci blok yu jëmmal, dagg fragmentation ak waste.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu KV Cache
Bi palanteer yi di tàmbalee ci téemeeri junni token, cache KV dafay nekk bottleneck bu mag, kon innovation bi dafa am doole: cache quantization ci 8 wala 4 bits, politiku dàq yuy daaneel token yu amul solo, cross-request prefix sharing, ak offloading ci CPU wala disk. Coppite ci architecture yu melni bàyyi xel ci bopp yu bari dañuy tënk cache bi ci boppam. Xaarandil wéyal co-design ci anam yu ñuy bàyyi xel ak sistemu mémoire yuñ fexe ngir liggéey ci contexte yu gudd lool ci njëg yu yomb ak ci produit yu bari.
Doxal ci àdduna dëgg
Gaawaale tontu chatbot ci jëfandikoowaat caabi/valeur yiñ denc ci jaar-jaaru waxtaan wi, moo gën ñu koy defaraat saa yu nekk.
Cache prefix biy séddoo cache bi ci sistem bu yàgg ci jëfandikukat yu bari, wàññi njëg ak latency.
vLLM's PagedAttention dafay yor cache KV ci ay blok ngir mëna def ay laaj yu bari ci benn GPU ci anam wu jaar yoon.
Quantizing cache KV ngir wàññi precision ngir mëna ànd ak contexte yu gëna gudd ci memory GPU bu néew.
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the KV Cache quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
KV Cache Optimisation
Laaj yi ñuy faral di laaj
What is KV Cache?
Cache KV dafay denc caabi ak vecteur valeur yi transformateur bi jota xayma ci token yu njëkk yi, kon jarul ñu xaymaat leen ci baat bu bees bumu defar. Mooy sabab bi gëna mag bi tax defar mbind gaaw - te li gëna am solo mooy lekk sa mémoire GPU ci jamonoy waxtaan yu yàgg.
Luy denc ci cache KV?
Cache KV dafay tëye caabi ak vecteur valeur yu bawoo ci jeton yu njëkk yi suko defee ñu mëna leen jëfandikoowaat ci barabu ordinatër.
Ban jafe-jafe la cache KV di njëkka saafara?
Bu cache amul, token bu bees bu nekk dafay laaj ñu xaymaat K/V ngir token bu njëkk bu nekk, te loolu amul benn njariñ; cache bi dafay tax njëgu token bu nekk nekk lu soppeeku.
Lan mooy njariñu cache KV?
Cache bi dafay màgg ak guddaayi sekans yi, diisaay yi, ak bopp yi, di faral di nekk konsomatër bi gëna am solo ci mémoire GPU ci sarwis yu yàgg yi.
Ban pexe mooy wàññi dayo cache KV ci séddoo caabi ak valeur ci boppu attention yi?
Laajte buñ boole ak laajte yu bari dañuy may njiitu laajte yu bari ñu séddoo ensemble caabi/valeur yu néew, di wàññi cache bi bu baax.
Lan mooy ñaari fase yi ci transformateur bi méngoo ak cache KV bi?
Prefill dafay feesal cache bi ak K/V bi ci laaj bi; decode dafay yokk benn K/V token bu bees jéego bu nekk ba noppi di topp cache bi.