KV cache
Ihe nchekwa KV na-echekwa igodo na uru vectors nke transformer agbakọọlarị maka akara ndị gara aga, yabụ na ọ gaghị agbakọ ha maka mkpụrụokwu ọhụrụ ọ bụla ọ na-ewepụta.
Nchịkọta
It is the single biggest reason text generation is fast — and the main thing eating your GPU memory during long conversations.
Ime miri emi
Ndị ntụgharị na-emepụta ederede otu token n'otu oge, na oyi akwa akara ngosi ọhụrụ ọ bụla kwesịrị iji tụnyere nke ọ bụla tupu akara. Usoro nlebara anya na-atụgharị akara ọ bụla ka ọ bụrụ ajụjụ, igodo, na vector uru. Na-enweghị caching, ịmepụta nọmba akara 1,000 ga-apụta iweghachi igodo na ụkpụrụ maka akara ngosi 999 mbụ n'oge ọ bụla - arụ ọrụ anọ na-abaghị uru. Ihe nchekwa KV na-echekwa igodo na uru vector ndị ahụ ka agbakọchara ha wee jiri ya mee ihe ọzọ, yabụ nzọụkwụ ọhụrụ ọ bụla na-agbakọ vectors maka otu akara ọhụrụ kachasị ọhụrụ wee gaa na ebe nchekwa echekwara. Nke a na-ebelata ọnụ ahịa otu-token site n'iji ogologo usoro were mee ka ọ bụrụ ihe na-adịgide adịgide. Azụmahịa ahụ bụ ebe nchekwa: cache na-eto n'ahịrị na ogologo okirikiri, ọnụ ọgụgụ nke ọkwa, na isi nlebara anya, na-aghọkarị onye na-azụ ahịa ebe nchekwa na-eje ozi ogologo oge.
Nghọta nka nka
N'oge usoro 'prefill' ihe nlereanya ahụ na-ahazi ngwa ngwa ma jupụta cache; N'oge 'decode' ọ na-agbakwunye otu K/V token kwa steepụ wee gụgharịa. Nha cache nha dị ka 2 (K na V) × n'ígwé × isi × isi_dim × usoro_ogologo × ogbe, na nkenke ahọpụtara. Iji mezi nke a, ụdị ọgbara ọhụrụ na-eji nleba anya nke mkpokọta ma ọ bụ ajụjụ ọtụtụ iji kesaa igodo/ụkpụrụ n'ofe isi, yana sistemụ ozi dị ka vLLM na-eji PagedAttention ekenye cache na ngọngọ na-abụghị nke na-aga n'ihu, na-egbutu nkewa na mkpofu.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke KV Cache
Dị ka windo ndị dị na ya na-agbatị n'ime narị puku kwuru puku akara, oghere KV na-aghọ ihe mgbochi etiti, yabụ ihe ọhụrụ dị oke egwu: ọnụọgụ cache na 8 ma ọ bụ 4 bits, atumatu nchụpụ na-agbadata akara ngosi dị ala, ịkekọrịta prefix arịrịọ, na ibuga na CPU ma ọ bụ diski. Nleba anya nleba anya nke isi ọtụtụ isi na-atụgharị cache n'onwe ya. Na-atụ anya na-aga n'ihu na-emekọ nhazi nke nlebara anya variants na usoro ebe nchekwa iji na-eje ozi ogologo ọnọdụ dị ọnụ ala na nnukwu mmepụta.
Mmejuputa n'ezie n'ụwa
Ịmelite nzaghachi chatbot ngwa ngwa site na iji igodo/ụkpụrụ echekwara na akụkọ mkparịta ụka kama ịhazigharị ya n'oge ọ bụla.
Ndozi ihe nrịbama nke na-ekekọrịta cache maka ogologo sistemụ na-aga n'ihu n'ọtụtụ ndị ọrụ, na-ebelata ọnụ ahịa yana latency.
vLLM's Paged Ntị n'ịhazi cache KV na ngọngọ iji jeere ọtụtụ arịrịọ oge na otu GPU nke ọma.
Na-agụta cache KV iji wedata nkenke ka ọ dabara ogologo ọnọdụ na ebe nchekwa GPU nwere oke.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the KV Cache quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
KV cache Optimization
Ajụjụ a na-ajụkarị
What is KV Cache?
Ihe nchekwa KV na-echekwa igodo na uru vectors nke transformer agbakọọlarị maka akara ndị gara aga, yabụ na ọ gaghị agbakọ ha maka mkpụrụokwu ọhụrụ ọ bụla ọ na-ewepụta. Ọ bụ otu nnukwu ihe kpatara ọgbọ ederede ji dị ngwa - yana isi ihe na-eri ebe nchekwa GPU gị n'oge mkparịta ụka ogologo.
Kedu ihe nchekwa nchekwa KV na-echekwa?
Ihe nchekwa KV na-ejide igodo na uru vectors sitere na akara ndị mbụ ka nlebara anya wee nwee ike iji ha ọzọ kama ịgbakọ ọzọ.
Kedu nsogbu cache KV na-edozi isi?
Na-enweghị caching, ọhụrụ token ọ bụla ga-achọ recomputing K/V maka ọ bụla gara aga akara, nke na-efunahụ; cache na-eme ka ọnụ ahịa otu-token na-eri ihe na-adịgide adịgide.
Kedu ihe bụ isi ala nke cache KV?
Ebe nchekwa ahụ na-eto n'usoro ogologo, akwa, na isi, na-abụkarị onye na-azụ ahịa ebe nchekwa GPU na-eje ozi ogologo oge.
Kedu usoro na-ebelata nha cache KV site na ịkekọrịta igodo na ụkpụrụ n'ofe isi nlebara anya?
Ajụjụ agbakọtara na nlebara anya ajụjụ dị iche iche na-eme ka ọtụtụ ndị isi ajụjụ kesaa obere igodo/uru tent, na-ebelata cache ahụ nke ukwuu.
Kedu akụkụ abụọ nke ntụgharị ntụgharị n'ihe gbasara cache KV?
Prefill jupụtara n'ime oghere na K/V ozugbo; decode na-agbakwunye otu K/V token ọhụrụ nzọụkwụ ọ bụla wee degharịa n'elu cache.