Ci bi mu melGis bi ci topp
KServe ak xeetu liggéey ci Kubernetes
Xarala
GUIDE teknik
Prefill ak decode buñ xaaj
Architecture buy liggéey buy xaaj modelu làkk bu yaatu ci ñaari fase yu wuute - prefill ak decode - ba noppi doxal leen ci pool yu wuute ci GPUs.
Déglul tegtal biiIA-nettali · ~2 min
Ci xët wii2 simili jàng
Résumé
Dafa am solo ndax ñaari fase yooyu dañu am bëgg-bëggu hardware bu wuute, te forse leen ñu nekk ci benn masin dafay yàq kàttan ba noppi gaañ latency.
Plongeur bu xóot
Su LLM tontu, dafay liggéey ci ñaari pàcc. Prefill dafay jàng laaj bi yépp benn yoon ba noppi tabax cache key-valeur (KV); lii ab burst bu rëy la, paralel, buñu tënk ci xayma buy feesal GPU bi ci math. Decode dafay defar ay token benn-benn, jéego bu nekk di jàng cache KV bi yépp - ab bandwidth bu memory, trickle bu woyof. Daw ñoom ñaar, ab prefill bu gudd dafay taxawal decode ku nekk (bopp-of-line blocking), ak batching ñaar yi defar ay jafe-jafe. Desagrégation dafay def prefill ci benn pool GPU ak decode ci beneen, toxal cache KV ci seen biir ci lëkkaloo yu gaaw yu melni NVLink wala InfiniBand. Pool bu nekk dañu koy ajuste ak scale moom boppam, gëna baaxal produit bi, gëna yombal latency geen gi, ba noppi may operatër yi ñu mëna dóor time-to-first-token ak time-per-output-token ci benn yoon.
Doxal ci àdduna dëgg
Assistant chat dafay yónnee dokimaa yu gudd yi ci cluster prefill bu bari ordinatër, ba noppi mu yónnee tontu yi ci cluster decode buñ defaree mémoire ngir mëna wéy di bind.
NVIDIA Dynamo ak vLLM may nañu operatër yi ñu tàqale prefill ak decode grupu liggéeykat yi suko defee ñu bari ay laaj yu gudd duñu gelé jikkoom yiy wéy.
Mooncake (Kimi bu Moonshot AI moo ko jëfandikoo) dafay xaaj prefill ak decode ba noppi yokk ci pool KV-cache buñ séddale ngir dagg xayma yu bari yi ci eskaal bi.
Ab sarwis buy yeggali kode dafay jagleel benn piscine bu ndaw ngir ay laaj yu gàtt ak benn piscine decode bu mag, ndax njëg yi gëna bari ñu ngi bawoo ci streaming ay token yu bari.
Ëlëgu prefill ak decode buñ xaaj
Xaarandil ni disagregation nekk na lu jaadu ci pilu liggéey yi. Sistem yu melni DistServe, Xaajale, ak Mooncake ñoo ko siiwal, vLLM ak NVIDIA Dynamo leegi dañuy yónnee ay anam yu wuute. Gëstu dafay puus KV-cache toxal gëna xéewale, cache pooling ak jëfandikoowaat ci laaj yi, ekilibre dinaamik bu prefill / decode ratios ci suufu dem bi ak dikk bi, ak boole bu gëna dëgër ak cache prefix ak prefill chunked. Lu palanteer yi di màgg ba nekk ay milioŋ ciy jeton, tàqale fase yooyu dafay gëna am solo ngir am xaalis bu bari te baax.
Teg ko ci jëf
Lu tax mu am solo
- Njëgg ak budget. Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
- dogal yu gëna leer. Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
- Xool kalite. Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Moytul
- Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
- Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
- Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Jéego yi ci topp
- Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
- Benchmark ci biir sargal ak done yu dëggu.
- Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
- Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Bulletin bu amul fayda
Jàngal xibaar AI bu bes
Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.
Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.
Nattal sa bopp
Take the Disaggregated Prefill and Decode Serving quiz
Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.
Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax
Laaj yi ñuy faral di laaj
Luy serwiis prefill ak decode buñ xaaj?
Architecture buy liggéey buy xaaj modelu làkk bu yaatu ci ñaari fase yu wuute - prefill ak decode - ba noppi doxal leen ci pool yu wuute ci GPUs. Dafa am solo ndax ñaari fase yooyu dañu am bëgg-bëggu hardware bu wuute, te forse leen ñu nekk ci benn masin dafay yàq kàttan ba noppi gaañ latency.
Lan mooy sabab bi tax ñuy tàqale prefill ak decode ci pool GPU yu wuute?
Prefill dafay doxal prompt bi yépp ci paralel ba noppi saturate calcul, ci noonu la decode di jàng cache KV jéego bu nekk te dafay yam ci bandwidth memory-appetites yu wuute yuy justifier pool yu wuute, yuñ defar seen bopp.
Ban jumtukaayu done lañu wara toxal ci liggéeykat yiy prefill dem ci liggéeykat yiy decode?
Prefill dafay tabax cache KV ngir laaj bi; decode mingi soxla cache bi ngir mëna wéy di defar, moo tax cache bi dafay jaar ci interconnexion bu gaaw ci pool decode bi.
Ban jafe-jafe la disaggregation di wàññi ci tabb GPU buñ bokk?
Ci GPU yiñ bokk, ab prefill bu gudd mën na gàllankoor jéego decode yiy wéy; tàqale leen dafay tere jafe-jafe yi ba noppi dakkal latency geen gi.
Lan moo waral prefill mën nañu ko batch bu baax waaye decode njariñu tuning bu wuute?
Prefill dafay boole jeton prompt yépp, suko defee lots yu gëna mag di dundal core tensor yi bu baax; decode dafay defar benn token benn yoon, te memory bi dafay tënk ko, moo tax dafay wuute ci eskaal yi.
Ban lëkkaloo lañuy gëna jëfandikoo ngir toxal cache KV ci diggante piscine yuñ xaajale?
Bandwidth bu rëy, lëkkalekaay yu néew latency yu melni NVLink (ci biir node) ak InfiniBand (ci biir node) lañu soxla suko defee toxal KV-cache du nekk bottleneck bu bees.
Weyal di jàng
Gid yu jëm ci loolu
Tann nañu yeneen njiit ngir topic bii