InoteveraGaidhi rinotevera
Kuvandudzwa kweKV Cache
Tekinoroji
Mutauro AI GUIDE
KV Cache (Kachengeta KV)
Iyo KV cache inochengetedza kiyi uye kukosha mavector shanduri yakatoverengera kune yakapfuura tokens, saka haifanirwe kuvadzokorodza kune yega yega izwi idzva raanogadzira.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Pa peji ino2 min verenga
Pfupiso
It is the single biggest reason text generation is fast — and the main thing eating your GPU memory during long conversations.
Kudzika Kwakadzika
Transformers inogadzira mavara tokeni imwe panguva, uye imwe neimwe itsva tokeni yekutarisisa inofanirwa kuenzaniswa neyakatangira tokeni. Iyo yekutarisisa meshini inoshandura chiratidzo chega chega kuita mubvunzo, kiyi, uye kukosha vector. Pasina caching, kugadzira nhamba yechiratidzo 1,000 zvinoreva kudzoreredza makiyi uye kukosha kune ese 999 ekutanga tokens padanho rega rega - quadratic, kutambisa basa. Iyo KV cache inochengetedza aya makiyi uye anokoshesa mavheji mushure mekunge atanga kuverengerwa uye kuashandisa zvakare, saka nhanho yega yega nyowani inongoverengera mavheji eiyo imwe nyowani tokeni uye inopinda pamusoro peyakachengetwa cache. Izvi zvinoderedza mutengo we-tokeni kubva pakuyera nehurefu hwekutevedzana kusvika pakungogara. Kutengeserana-yekuyeuka: iyo cache inokura zvakatevedzana nehurefu hwemamiriro, huwandu hwematanho, uye misoro yekutarisisa, kazhinji ichive inotungamira ndangariro mutengi mukureba-mamiriro ekushandira.
Kuitwa muNyika yeChokwadi
Kumhanyisa mhinduro dzechatbot nekushandisa zvakare makiyi akavharirwa / kukosha kubva munhoroondo yehurukuro pane kuidzokororazve.
Prefix caching inogovera cache yenguva refu yekukurumidza kune vakawanda vashandisi, kucheka mutengo uye latency.
vLLM's PagedAttention inobata KV cache mumabhuroko kuti ishumire zvikumbiro zvakawanda panguva imwe chete pane imwe GPU zvakanaka.
Kuverengera iyo KV cache kudzikisa iko kurongeka kuti ikwane kureba mamiriro mune shoma GPU ndangariro.
Ramangwana reKV Cache
Sezvo mahwindo emukati anotambanudzira mumazana ezviuru zvezviratidzo, iyo KV cache inova iyo yepakati bhodhoro, saka hutsva hunotyisa: cache quantization kune 8 kana 4 bits, mitemo yekudzinga inodonhedza yakaderera-yakakosha tokens, kuyambuka-chikumbiro prefix kugovera, uye kurodha kuCPU kana diski. Architectural shifts senge multi-head latent attention inomanikidza cache pachayo. Tarisira kuenderera mberi dhizaini yekutarisisa akasiyana uye ndangariro masisitimu ane chinangwa chekushandira akareba mamiriro akachipa uye nepamusoro pekupinda.
Kuzvishandisa
Nei zvichikosha
- Kumhanya uye chiyero. Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana.
- Svika uye svika. Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana.
- Sarudzo dzakajeka. Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora.
Chenjera
- Chokwadi chehuroyi chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana tsvakiridzo.
- Kunzwa nekukasira kunogona kugadzira mhedzisiro isingaenderane pane zvikumbiro zvakafanana.
- Sensitive text data inogona kuburitswa kana zvidhiraivho zvisina kusimba.
Matanho anotevera
- Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa.
- Mhinduro dzepasi neakavimbika masosi pese pazvine basa.
- Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda.
- Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva.
Ramba Uchiongorora
Tsamba yemahara
Tora iyo yemazuva ese AI pfupiso
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Take the KV Cache quiz
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Mibvunzo inowanzo bvunzwa
What is KV Cache?
Iyo KV cache inochengetedza kiyi uye kukosha mavector shanduri yakatoverengera kune yakapfuura tokens, saka haifanirwe kuvadzokorodza kune yega yega izwi idzva raanogadzira. Ndicho chikonzero chimwe chete chikuru chikonzero chizvarwa chemavara chinokurumidza - uye chinhu chikuru chiri kudya ndangariro yako yeGPU panguva yehurukuro refu.
Ko KV cache inochengeterei?
Iyo KV cache inobata kiyi uye kukoshesa mavheji kubva kune ekutanga tokens saka kutarisisa kunogona kuvashandisa zvakare pane kuverengera.
Nderipi dambudziko iro KV cache inonyanya kugadzirisa?
The cache reuses keys and values computed for earlier tokens instead of rebuilding them at every decode step. The new query still attends over cached positions, so attention work and cache memory continue to grow with context length.
Chii chakanyanya kuderera cheKV cache?
Iyo cache inokura nekutevedzana kureba, maseru, uye misoro, kazhinji ichiva mutengi weGPU ndangariro mukureba-mamiriro ekushandira.
Ndeipi nzira inoderedza saizi yeKV cache nekugovana makiyi uye kukosha pamisoro yekutarisisa?
Mumapoka-mubvunzo uye kutarisisa kwemibvunzo yakawanda regai misoro yemibvunzo yakawanda igovane mashoma makiyi / kukosha seti, ichideredza cache zvakanyanya.
Ndeapi zvikamu zviviri zveshanduro yetransformer inoenderana neKV cache?
Prefill inozadza cache neiyo K/V yekukurumidza; decode inowedzera imwe tokeni itsva K/V nhanho imwe neimwe uye inodzokorora pamusoro pecache.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi