xët yi ak vLLM
PagedAttention ab pexem yoriinu mémoire la buy denc cache attention bu modelu làkk ci ay bloc yu ndaw yuñ mëna jëfandikoowaat ci barabu benn pàcc bu mag bu toppalante.
Résumé
It powers vLLM, an open-source serving engine that dramatically boosts how many requests a single GPU can handle.
Plongeur bu xóot
Su benn xeetu làkk defaree mbind, dafay denc 'KV cache' (vecteur key ak valeur) ngir bépp token bumu gis suko defee token bi ci topp mëna topp contexte bi yépp. Ci cosaan, laaj bu nekk dafa daan denc benn daal bu yaatu ci memory GPU buñ dajale ci guddaay bi gëna mag, di yàq xaalis bu bari sudee toppalante yi dañu gëna gàtt wala ñu wuute ci guddaay. PagedAttention, biñ dugal ci këyitu vLLM 2023 bu bawoo UC Berkeley, dafa leble xalaatu paging mémoire virtuel ci sistem operaasioŋ yi: dafay xaaj cache KV ci ay blok yu am dayo bu mën dëkk fépp ci mémoire bi, ñu mëna ko jox suñu ko laajee. Tablo seetlu dafay wane barabi jeton logic yi ci blok physique yi. Loolu daanaka dina dindi xaajalug mémoire bi ba noppi may ñu séddoo ay blok, ci misaal ci ay génn yu bari ci benn laaj.
Gis-gis xarala
Cache KV dafa xaajaloo ay xët yu am dayo buñ tëral, bu nekk ci ñoom yor caabi yi ak valeur yi ngir benn limu jeton yuñ tànn. Tablo blok bu toppalante bu nekk dafay màndargaal barabi logic yi ak barabi xët physique yi, kon cache bu toppalante bi warul nekk ci diggante. Ndax prefix yu nuróo (ab sistem buñ bokk, wala car yiy seetlu ci limyéer) mën nañu joxoñ benn xët physique jaaraleko ci kopie-ci-bind, dañuy jëfandikoowaat mémoire bi ci barabu ñaari yoon, dagg mbalit mi ci lu ëpp 60% ba ci yenn pursaa.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu PagedAttention ak vLLM
vLLM nekk na yaxu ndigg lu ubbeeku, te xalaati PagedAttention leegi dañuy feeñ ci stack yiy liggéey. Xaarandi cache prefix bu gëna xóot (jëfandikuwaat sistem cache bi laaj ci jëfandikukat yi), prefill buñ xaaj ak dekode ci masin yu wuute, politiku dàq yu gëna am xel, ak boole bu dëgër ak kantite ak dekode speculatif. Lu palanteer yi di màgg ba nekk ay milioŋ ciy jeton, njiitu KV paged bu baax dafay gëna am solo ngir mëna wéy di liggéey ci njëg yu woyof.
Doxal ci àdduna dëgg
Hosting LLM API open-source fu vLLM di liggéeyal jëfandikukat yu bari yuy waxtaan ci benn GPU ci produit bu bari
Séddoo ab sistem bu gudd ci ay junni laaj jaaraleko ci prefix cache suko defee ñu def ko benn yoon, te baña baamtu
Seetug beam buy daw wala ay misaal yu bari yuy séddoo ay bloku KV ngir laaj buñ bokk jaaraleko ci kopie-ci-bind
Dagg mbalitu mémoire GPU ci xaaj-xaaj suko defee ab joxekat mëna def ay sesioŋ yu bari ci benn aparey bi
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the PagedAttention and vLLM quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Njaboot gu wuute
Laaj yi ñuy faral di laaj
What is PagedAttention and vLLM?
PagedAttention ab pexem yoriinu mémoire la buy denc cache attention bu modelu làkk ci ay bloc yu ndaw yuñ mëna jëfandikoowaat ci barabu benn pàcc bu mag bu toppalante. Dafay doxal vLLM, benn motër buy liggéey ci open-source buy yokk bu baax limu laaj yi benn GPU mëna def.
Lan la 'KV cache' di denc ci defar mbind?
Cache KV dafay tëye vecteur key ak valeur ngir bépp token bu njëkk, loolu mooy tax model bi mëna topp contexte bi yépp te doo ko xaymaat jéego bu nekk.
Ban xalaatu sistem jëfandikoo moo inspiré PagedAttention?
PagedAttention dafay leble paging ci mémoire virtuel: cache KV dafa xaajaloo ci xët yu am dayo bu takku bu mëna dëkk fépp, ñu boole ci tablo blok.
Lan mooy jafe-jafe bi PageAttention mëna saafara ci wàllu cache KV?
Reserve benn slab bu rëy bu jege ci laaj bu nekk, dayo ngir guddaay bi gëna mag, yàq na memory GPU bu bari. Paging dafay xaaj ay bloc yu ndaw ci laaj, dagg mbalit mi.
naka lay def ba PagedAttention may génne yu bari ñu séddoo mémoire ngir benn laaj buñ bokk?
Prefix yu nuróo (prompt yuñ bokk wala car yiy seetlu ci limyéer) dañuy joxoñ benn xëtu KV physique, ñuy kopie ludul su benn car diverge.
Fan lañu njëkka defar vLLM ak PagedAttention?
vLLM ak algorithmu PagedAttention génnewoon nañu ci UC Berkeley ci benn këyit ci 2023, ci diir bu gàtt mu nekk motëru serwiis bu ñuy jëfandikoo bu baax.