Imọ Itọsọna
Imudara Kaṣe KV
Kaṣe KV n tọju awọn bọtini ati iye ti oluyipada kan ti ṣe iṣiro tẹlẹ nitorina ko tun ṣiṣẹ fun gbogbo ami tuntun - ṣugbọn o le balloon si gigabytes.
Gbọ si itọsọna yiiAI-ṣe afihan · ~2 min
Lori iwe yi2 min ka
Akopọ
Iṣapeye kaṣe KV dinku ati ṣakoso iranti yẹn nitorinaa awọn awoṣe sin awọn ipo gigun si awọn olumulo diẹ sii ni ẹẹkan.
Jin Dive
In a transformer, each new token attends to all previous tokens via attention's keys (K) and values (V). Recomputing K and V for the whole sequence at every step would be quadratic and wasteful, so models cache them: the KV cache. The downside is size. The cache grows linearly with sequence length, batch size, layers, and heads, so a long-context request can consume more GPU memory than the model weights themselves. Optimization tackles this from several angles: paged memory (vLLM's PagedAttention) stores the cache in non-contiguous blocks to reduce fragmentation and enable sharing; quantization stores K and V in 8-bit or 4-bit; and architectural changes like Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) let many query heads share fewer key/value heads, slashing cache size at the source.
Real-World imuse
vLLM's PagedAttention serving many concurrent chat sessions by packing KV blocks while reducing memory fragmentation
Ifarabalẹ-Ibeere akojọpọ ni awọn awoṣe Llama ti o dinku iwọn kaṣe KV nitoribẹẹ awọn ipo to gun ni ibamu si iranti GPU
Didiwọn kaṣe KV si 8-bit (KV8) lati dinku iranti kaṣe ni aijọju lakoko akopọ iwe-gigun
Iṣaju iṣaju ti o tun lo awọn bulọọki KV ti itọsi eto pinpin kọja ẹgbẹẹgbẹrun awọn ibeere API
Ojo iwaju ti KV Cache Iṣapeye
Bi awọn window ti o tọ ti na si awọn ọgọọgọrun egbegberun tabi awọn miliọnu awọn ami-ami, kaṣe KV di idiyele ti o ga julọ ti iṣẹ. Reti funmorawon kaṣe ibinu ati itusilẹ (sisọ awọn ami akiyesi kekere silẹ), pinpin ibeere ibeere agbelebu bi aiyipada, gbigbe kaṣe tutu si Sipiyu tabi NVMe, ati awọn faaji bii MLA ati GQA di boṣewa. Ṣiṣakoso kaṣe yoo pọ si ni afiwe awọn ilana iranti ni kikun pẹlu awọn ipele ati iṣaju ọlọgbọn.
Fifi o sinu iwa
Kini idi ti o ṣe pataki
- Iye owo ati isuna. Awọn ipinnu faaji ṣe awakọ iṣẹ ati idiyele iṣẹ fun awọn ọdun.
- Awọn ipinnu diẹ sii. Ẹkọ imọ-ẹrọ ṣe iranlọwọ fun awọn ẹgbẹ lati yan akopọ to tọ, kii ṣe ọkan tuntun nikan.
- Iṣakoso didara. Awọn yiyan imọ-ẹrọ to dara julọ dinku awọn iṣẹlẹ igbẹkẹle ni iṣelọpọ.
Ṣọra fun
- Ṣiṣepe ala-ilẹ kan le tọju awọn ailagbara eto ti o gbooro.
- Awọn ohun elo amayederun ati awọn idiyele itọju nigbagbogbo ni aibikita.
- Aabo ati awọn ela akiyesi le dagba bi awọn eto ṣe di eka sii.
Next awọn igbesẹ
- Ṣetumo lairi, didara, ati awọn ibi-afẹde idiyele ṣaaju imuse.
- Aṣepari labẹ ẹru ojulowo ati awọn ipo data.
- Abojuto ohun elo fun awọn aṣiṣe, fiseete, ati ipa olumulo.
- Mura ipadasẹhin pada ati awọn ipa ọna esi iṣẹlẹ ṣaaju iwọn.
Tesiwaju Ṣiṣawari
Ìwé ìròyìn ọfẹ
Gba ìkànìyàn AI ojoojúmọ́
Ìtàn AI mẹta tó jẹ́rìí ní gbogbo owurọ́ ọjọ́ iṣẹ́, kọ́wé ní Gẹ̀ẹ́sì tó rọrùn. Ọfẹ́ fún gbogbo àkókò, kò sí ìpolówó.
Ìmeèlì kan ní gbogbo ọjọ́ iṣẹ́. Yọkúrò ní ìkànìyàn ní ìkànìyàn kan. A kò ta tàbí pín àdírẹ́sì rẹ.
Ṣe idanwo fun ara rẹ
Take the KV Cache Optimization quiz
Ìdáhùn lẹ́sẹ̀kẹsẹ̀ lórí gbogbo ìdáhùn, àti iwe-ẹ̀rí tó lè pin pẹ̀lú ID tó lè jẹ́rìí lẹ́yìn tí o bá kọ́ ẹ̀kọ́.
Ṣe àtìlẹ́yìn ẹ̀kọ́ AI ọfẹ́. AI Understanding jẹ́ àjọ àìní èrè 501(c)(3) — kò sí ìpolówó, kò sí àkúnya owó, ní gbogbo àkókò. Ṣe ẹbun kan
Awọn ibeere ti a beere nigbagbogbo
Kini KV Cache Optimization?
Kaṣe KV n tọju awọn bọtini ati iye ti oluyipada kan ti ṣe iṣiro tẹlẹ nitorina ko tun ṣiṣẹ fun gbogbo ami tuntun - ṣugbọn o le balloon si gigabytes. Imudara kaṣe KV dinku ati ṣakoso iranti yẹn ki awọn awoṣe ṣe iranṣẹ awọn ipo gigun si awọn olumulo diẹ sii ni ẹẹkan.
Kini ibi ipamọ KV ati kilode?
Awọn bọtini caching ati awọn iye lati awọn ami ami iṣaaju yago fun atunṣe iṣiro akiyesi lori gbogbo ami tuntun, fifipamọ akoko.
Kini idi ti kaṣe KV le di iṣoro iranti?
Awọn iwọn iwọn kaṣe pẹlu ipari ọrọ-ọrọ ati ibaramu, nitorinaa awọn ibeere gigun le lo awọn oye pupọ ti iranti GPU.
Ero-ero ẹrọ ṣiṣe wo ni PagedAttention yawo?
Paged Ifarabalẹ tọju kaṣe naa sinu awọn bulọọki iwọn ti o wa titi ti kii-contiguous ti a ya aworan nipasẹ tabili kan, gẹgẹ bi paging OS.
Bawo ni Ibeere Ikojọpọ ati Ifarabalẹ Ọpọ-Ibeere ṣe dinku iwọn kaṣe KV?
Pínpín bọtini/awọn olori iye kọja awọn ori ibeere pupọ tumọ si diẹ diẹ K ati awọn fekito V lati fipamọ.
Kini anfani kan ti pinpin iṣaaju ninu kaṣe KV?
Itọkasi eto pinpin ṣe agbejade awọn titẹ sii KV kanna, nitorinaa awọn ibeere lọpọlọpọ le tọka si awọn bulọọki kanna dipo ṣiṣe ẹda wọn.
Tesiwaju kikọ
Jẹmọ awọn itọsọna
Awọn itọsọna diẹ sii ti a yan fun koko yii