KV Cache
Ubwihisho bwa KV bubika urufunguzo nagaciro kerekana impinduka transformateur yamaze kubara kubimenyetso byabanje, ntabwo rero igomba kubisubiramo kuri buri jambo rishya ritanga.
Incamake
It is the single biggest reason text generation is fast — and the main thing eating your GPU memory during long conversations.
Kwibira cyane
Abahinduzi batanga inyandiko ikimenyetso kimwe icyarimwe, kandi buri kimenyetso gishya cyitonderwa kigomba kugereranya na buri kimenyetso kibanziriza. Uburyo bwo kwitondera buhindura buri kimenyetso mubibazo, urufunguzo, nagaciro kerekana. Hatabayeho gufunga, kubyara ikimenyetso 1.000 bisobanura kubara urufunguzo nagaciro kubimenyetso 999 byambere kuri buri ntambwe - umurimo wa quadratic, usesagura. Ubwihisho bwa KV bubika urufunguzo nagaciro byagaciro nyuma yo kubarwa bwa mbere no kubikoresha, buri ntambwe nshya rero ibara ibice gusa kubimenyetso bishya kandi ikitabira hejuru ya cache yabitswe. Ibi bigabanuka kuri buri kimenyetso kuva gupima hamwe nuburebure bwikurikiranya kugeza hafi. Ubucuruzi-bwibukwa ni kwibuka: cache ikura neza hamwe nuburebure bwimiterere, umubare wibice, hamwe nimitwe yitabwaho, akenshi biba byiganjemo abaguzi bibuka mugihe kirekire.
Ubushishozi
Mugihe cya 'prefill' icyiciro icyitegererezo gitunganya ikibazo cyose kandi cyuzuza cache; mugihe cya 'decode' yongeraho ikimenyetso kimwe K / V kuri buri ntambwe hanyuma ikongera. Ingano yubunini bwa 2 (K na V) × ibice × imitwe × umutwe_dim × ikurikiranye_uburebure × icyiciro, muburyo bwatoranijwe. Kugirango ukemure ibi, moderi zigezweho zikoresha amatsinda-kubaza cyangwa kubaza ibibazo byinshi kugirango dusangire urufunguzo / indangagaciro mumitwe, hamwe na sisitemu nka vLLM ikoresha PagedIbyitonderwa kugirango igabanye cache mubice bidahuza, guca ibice hamwe n imyanda.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza ka KV Cache
Mugihe imiterere ya Windows irambuye mu bihumbi magana, ibimenyetso bya KV bihinduka icyuho cyo hagati, bityo guhanga udushya birakaze: kubara cache kugeza kuri 8 cyangwa 4 bits, politiki yo kwirukana igabanya ibimenyetso bidafite akamaro gake, kugabana ibyifuzo mbere yo kugabana, no gupakurura CPU cyangwa disiki. Imyubakire ihindagurika nkimitwe myinshi yihishe yitonze ikanda cache ubwayo. Tegereza gukomeza gufatanya gushushanya ibitekerezo hamwe na sisitemu yo kwibuka igamije gukora ibintu birebire cyane bihendutse kandi byinjira cyane.
Gushyira mu bikorwa Isi
Kwihutisha chatbot isubiza mukoresheje urufunguzo rwihishwa / indangagaciro kuva mumateka y'ibiganiro aho kuyisubiramo buri gihe.
Prefix cishing isangira cache kumurongo muremure wa sisitemu kubakoresha benshi, kugabanya ibiciro nubukererwe.
vLLM's PagedIcyifuzo cyo gucunga KV cache mubice kugirango ukore ibyifuzo byinshi bihurira kuri GPU imwe neza.
Kugabanya ubwihisho bwa KV kugirango ugabanye neza kugirango uhuze umwanya muremure mububiko buke bwa GPU.
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the KV Cache quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
KV Cache Optimisation
Ibibazo bikunze kubazwa
What is KV Cache?
Ubwihisho bwa KV bubika urufunguzo nagaciro kerekana impinduka transformateur yamaze kubara kubimenyetso byabanje, ntabwo rero igomba kubisubiramo kuri buri jambo rishya ritanga. Ninimpamvu imwe nini ituma inyandiko yihuta - kandi ikintu nyamukuru urya ububiko bwa GPU mugihe cyibiganiro birebire.
Ububiko bwa KV bubika iki?
Ubwihisho bwa KV bufite urufunguzo nagaciro kerekana ibimenyetso kuva ibimenyetso byambere kugirango ubwitonzi bushobora kongera kubikoresha aho kubara.
Ni ikihe kibazo cache ya KV ikemura cyane cyane?
Hatabayeho gufunga, buri kimenyetso gishya gisaba kubara K / V kuri buri kimenyetso cyabanjirije iki, cyangiza; cache ikora kuri buri kimenyetso igiciro gihoraho.
Ni ubuhe butumwa bukomeye bwa cache ya KV?
Ubwihisho bukura hamwe nuburebure bwikurikiranya, ibice, n imitwe, akenshi biba umuguzi wiganjemo ububiko bwa GPU murwego rurerure rukorera.
Nubuhe buryo bugabanya ubunini bwa KV mugusangira urufunguzo nagaciro mumitwe yitabwaho?
Itsinda-ryabajijwe hamwe nibibazo byinshi byitondewe reka reka imitwe myinshi yibibazo isangire bike urufunguzo / agaciro gashizweho, kugabanya cache cyane.
Nibihe byiciro bibiri byerekana impinduka ugereranije na KV cache?
Kuzuza kuzuza cache hamwe na K / V; decode yongeramo ikimenyetso gishya K / V buri ntambwe hanyuma igaruka hejuru ya cache.