Ururimi AI

Cache ya KV

Ubwihisho bwa KV bubika urufunguzo nagaciro kerekana impinduka transformateur yamaze kubara kubimenyetso byabanje, ntabwo rero igomba kubisubiramo kuri buri jambo rishya ritanga.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Kazoza ka KV Cache
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

It is the single biggest reason text generation is fast — and the main thing eating your GPU memory during long conversations.

Kwibira cyane

Abahinduzi batanga inyandiko ikimenyetso kimwe icyarimwe, kandi buri kimenyetso gishya cyitonderwa kigomba kugereranya na buri kimenyetso kibanziriza. Uburyo bwo kwitondera buhindura buri kimenyetso mubibazo, urufunguzo, nagaciro kerekana. Hatabayeho gufunga, kubyara ikimenyetso 1.000 bisobanura kubara urufunguzo nagaciro kubimenyetso 999 byambere kuri buri ntambwe - umurimo wa quadratic, usesagura. Ubwihisho bwa KV bubika urufunguzo nagaciro byagaciro nyuma yo kubarwa bwa mbere no kubikoresha, buri ntambwe nshya rero ibara ibice gusa kubimenyetso bishya kandi ikitabira hejuru ya cache yabitswe. Ibi bigabanuka kuri buri kimenyetso kuva gupima hamwe nuburebure bwikurikiranya kugeza hafi. Ubucuruzi-bwibukwa ni kwibuka: cache ikura neza hamwe nuburebure bwimiterere, umubare wibice, hamwe nimitwe yitabwaho, akenshi biba byiganjemo abaguzi bibuka mugihe kirekire.

Gushyira mu bikorwa Isi

Kwihutisha chatbot isubiza mukoresheje urufunguzo rwihishwa / indangagaciro kuva mumateka y'ibiganiro aho kuyisubiramo buri gihe.

Prefix cishing isangira cache kumurongo muremure wa sisitemu kubakoresha benshi, kugabanya ibiciro nubukererwe.

vLLM's PagedIcyifuzo cyo gucunga KV cache mubice kugirango ukore ibyifuzo byinshi bihurira kuri GPU imwe neza.

Kugabanya ubwihisho bwa KV kugirango ugabanye neza kugirango uhuze umwanya muremure mububiko buke bwa GPU.

Kazoza ka KV Cache

Mugihe imiterere ya Windows irambuye mu bihumbi magana, ibimenyetso bya KV bihinduka icyuho cyo hagati, bityo guhanga udushya birakaze: kubara cache kugeza kuri 8 cyangwa 4 bits, politiki yo kwirukana igabanya ibimenyetso bidafite akamaro gake, kugabana ibyifuzo mbere yo kugabana, no gupakurura CPU cyangwa disiki. Imyubakire ihindagurika nkimitwe myinshi yihishe yitonze ikanda cache ubwayo. Tegereza gukomeza gufatanya gushushanya ibitekerezo hamwe na sisitemu yo kwibuka igamije gukora ibintu birebire cyane bihendutse kandi byinjira cyane.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Umuvuduko n'igipimo. Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
  • Kugera no kugera. Yagura uburyo bwindimi nuburyo bwo gutumanaho.
  • Ibyemezo bisobanutse. Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.

Witondere

  • Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
  • Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
  • Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.

Intambwe ikurikira

  1. Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
  2. Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
  3. Komeza kugenzura abantu kugenzura ibisubizo byinshi.
  4. Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the KV Cache quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is KV Cache?

Ubwihisho bwa KV bubika urufunguzo nagaciro kerekana impinduka transformateur yamaze kubara kubimenyetso byabanje, ntabwo rero igomba kubisubiramo kuri buri jambo rishya ritanga. Ninimpamvu imwe nini ituma inyandiko yihuta - kandi ikintu nyamukuru urya ububiko bwa GPU mugihe cyibiganiro birebire.

Ububiko bwa KV bubika iki?

Ubwihisho bwa KV bufite urufunguzo nagaciro kerekana ibimenyetso kuva ibimenyetso byambere kugirango ubwitonzi bushobora kongera kubikoresha aho kubara.

Ni ikihe kibazo cache ya KV ikemura cyane cyane?

The cache reuses keys and values computed for earlier tokens instead of rebuilding them at every decode step. The new query still attends over cached positions, so attention work and cache memory continue to grow with context length.

Ni ubuhe butumwa bukomeye bwa cache ya KV?

Ubwihisho bukura hamwe nuburebure bwikurikiranya, ibice, n imitwe, akenshi biba umuguzi wiganjemo ububiko bwa GPU murwego rurerure rukorera.

Nubuhe buryo bugabanya ubunini bwa KV mugusangira urufunguzo nagaciro mumitwe yitabwaho?

Itsinda-ryabajijwe hamwe nibibazo byinshi byitondewe reka reka imitwe myinshi yibibazo isangire bike urufunguzo / agaciro gashizweho, kugabanya cache cyane.

Nibihe byiciro bibiri byerekana impinduka ugereranije na KV cache?

Kuzuza kuzuza cache hamwe na K / V; decode yongeramo ikimenyetso gishya K / V buri ntambwe hanyuma igaruka hejuru ya cache.