KV cache
Ma'ajiyar KV tana adana maɓalli da ƙima na kayan aikin da taswirar ta riga ta ƙididdige su don alamun da suka gabata, don haka ba dole ba ne a sake ƙididdige su ga kowace sabuwar kalma da ta haifar.
Dubawa
It is the single biggest reason text generation is fast — and the main thing eating your GPU memory during long conversations.
Zurfafa nutsewa
Masu canza canji suna samar da rubutu alama ɗaya a lokaci guda, kuma kowane sabon layin alamar alama yana buƙatar kwatanta da kowane alamar da ta gabata. Tsarin kulawa yana juya kowace alama zuwa tambaya, maɓalli, da ƙima. Ba tare da caching ba, samar da lambar alama 1,000 na nufin sake lissafin maɓallai da ƙima ga duk alamun 999 na farko a kowane mataki - muƙamai, aikin ɓarna. Cache na KV yana adana waɗannan maɓalli da ƙima bayan an fara ƙididdige su kuma sake amfani da su, don haka kowane sabon mataki kawai yana ƙididdige ƙididdiga don sabuwar alama guda ɗaya kuma yana halartar wurin da aka adana. Wannan yana rage farashin kowane-alami daga ƙima tare da tsayin jeri zuwa kusan akai. Kasuwancin kashewa shine ƙwaƙwalwar ajiya: cache yana girma a layi tare da tsayin mahallin, adadin yadudduka, da kawunan hankali, sau da yawa ya zama babban mabukaci ƙwaƙwalwar ajiya a cikin sabis na dogon lokaci.
Fahimtar Fasaha
A lokacin 'prefill' tsarin samfurin yana aiwatar da duk abin da sauri kuma ya cika cache; yayin 'decode' yana haɗa alamar K/V ɗaya a kowane mataki kuma ya sake karantawa. Ma'auni girman cache kamar 2 (K da V) × yadudduka × kawuna × head_dim × jerin_tsawon tsari × tsari, a cikin madaidaicin zaɓi. Don haɓaka wannan, ƙirar zamani suna amfani da tarin tambaya-tambaya ko kulawar tambaya da yawa don raba maɓallai/daraja a cikin kawuna, da tsarin sabis kamar vLLM suna amfani da PagedAttention don ware cache a cikin ɓangarorin da ba su da alaƙa, yanke rarrabuwa da sharar gida.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar KV Cache
Kamar yadda mahallin windows ke shimfiɗa cikin ɗaruruwan dubunnan alamu, cache na KV ya zama babban ƙulli, don haka ƙididdigewa yana da zafi: ƙididdige cache zuwa rago 8 ko 4, manufofin korar da ke sauke alamun ƙarancin mahimmanci, raba buƙatun prefix, da saukewa zuwa CPU ko faifai. Canza gine-gine kamar latent hankali da yawa damtse cache kanta. Yi tsammanin ci gaba da ƙira na bambance-bambancen hankali da tsarin ƙwaƙwalwar ajiya da nufin yin hidimar dogayen mahallin cikin rahusa kuma a babban kayan aiki.
Aiwatar da Gaskiyar Duniya
Haɓaka amsa chatbot ta hanyar sake amfani da maɓalli/daraja da aka ɓoye daga tarihin tattaunawa maimakon sake sarrafa shi kowane juzu'i.
Prefix caching wanda ke raba cache don dogon tsarin faɗakarwa a tsakanin masu amfani da yawa, yanke farashi da latency.
vLLM's Paged Hankali yana sarrafa ma'ajiyar KV a cikin tubalan don ba da buƙatun lokaci ɗaya akan GPU ɗaya da inganci.
Ƙididdiga cache na KV zuwa ƙananan daidaito don dacewa da tsayin daka cikin ƙayyadaddun ƙwaƙwalwar GPU.
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the KV Cache quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
KV Cache Ingantawa
Tambayoyin da ake yawan yi
What is KV Cache?
Ma'ajiyar KV tana adana maɓalli da ƙima na kayan aikin da taswirar ta riga ta ƙididdige su don alamun da suka gabata, don haka ba dole ba ne a sake ƙididdige su ga kowace sabuwar kalma da ta haifar. Shi ne babban dalilin da yasa tsara rubutun ke da sauri - kuma babban abin cin ƙwaƙwalwar GPU ɗin ku yayin tattaunawa mai tsawo.
Menene KV cache ke adanawa?
Cache na KV yana riƙe da maɓalli da ƙima daga alamun farko don haka hankali zai iya sake amfani da su maimakon yin lissafi.
Wace matsala ce KV cache ke magance da farko?
Ba tare da caching ba, kowane sabon alamar yana buƙatar sake lissafin K/V don kowane alamar da ta gabata, wanda ke ɓarna; cache yana sa farashin kowane-token ya yi kusan akai.
Menene babban abin da ke cikin cache na KV?
Cache yana girma tare da tsayin jeri, yadudduka, da kawunan kai, galibi ya zama babban mabukaci na ƙwaƙwalwar GPU a cikin sabis na dogon lokaci.
Wace dabara ce ke rage girman cache na KV ta hanyar raba maɓalli da ƙima a cikin kawunan hankali?
Ƙungiya-tambayi da kulawar tambaya da yawa suna barin shugabannin tambaya da yawa su raba ƴan maɓalli/ƙimar ƙima, suna raguwar cache sosai.
Wadanne nau'ikan nau'ikan nau'ikan na'urori guda biyu ne na tantancewar taswira dangane da cache na KV?
Prefill ya cika cache tare da K/V na gaggawa; yanke hukunci yana ƙara sabon alamar K/V kowane mataki kuma yana sake karanta cache.